Multiple models exist for the evaluation of human motor performance; some of these rely on the Index of Difficulty (ID), a measure to evaluate the difficulty associated to simple reaching tasks. Despite the numerous applications of the ID in reaching movements, the existing formulations are functions of the geometrical features of the task and do not consider the motor behaviour of subjects performing repetitive movements in interaction with the environment. Variability of movements, length of trajectories, subject-specific strength and skill, and required interaction with the environment are all factors that contribute to the motor difficulty experienced by a moving agent  as it repeatedly interacts with the environment during a given task . A novel concept of motor difficulty experienced by an agent executing repetitive end-effector movements is presented in this study. A stochastic ID formulation is proposed that captures the abovementioned factors and applies to general three-dimensional motor tasks. Natural motor variability, inherent in the proposed model, is representative of the flexibility in motor synergies for a given agent-environment interaction: the smaller the flexibility, the greater the experienced difficulty throughout the movement. The quantification of experienced motor difficulty is demonstrated for the case of young healthy subjects performing three-dimensional arm movements during which different objects are manipulated. Results show that subjects’ experienced motor difficulty is influenced by the type of object. In particular, a difference in motor difficulty is observed when manipulating objects with different grasp types. The proposed model can be employed as a novel tool to evaluate the motor performance of agents involved in repetitive movements, such as in pick and place and manipulation, with application in both industrial and rehabilitation contexts. The evaluation of human motor performance is a topic of interest in multiple applications, ranging from manual operations in a production line, to human-computer interface, and human biomechanics and rehabilitation. In these fields, movements such as reaching and pick and place are most commonly studied from the perspectives of motor  and cID)  at a probability level s of the n points qk(s) with probability level φ is converted into a circular area of radius rmean. This average radius is used in this study to model the stochastic width associated with the repetitive movement, as follows:With this integral, the elliptical area that characterizes the spatial distribution at P(s) can be captured by the single value of the stochastic width Wobs. The practical use of the above model of stochastic width Wobs in capturing the motor variability will be shown in the following sections. Nevertheless, other models for stochastic width could be assumed, without losing generality in the proposed analytical-stochastic framework for the evaluation of motor difficulty.Through this model, the information of the observed end-effector spatial variability on the plane IDobs) to capture the effects of different external constraints on an agent’s experienced difficulty is demonstrated through the analysis of repetitive movements in the execution of a three-dimensional manipulation task. Results of the statistical analysis on IDobs values are provided and discussed.The application of the proposed model of difficulty . Data refer to 31 healthy individuals; all subjects belong to the same population of young healthy adults and, therefore, no between-subjects differences are assumed. Each individual sits in front of a table to perform the manipulation task with the dominant hand. The initial object position is placed 300 mm from the edge of the table on the same side of the dominant hand; here, the subject grasps an object and moves it to the release position, placed 500 mm symmetrically on the other side of the table, where the object is released and, finally, repositions the hand at the starting point. For the experiment, five different objects have been considered: a tennis ball (105 mm diameter), two one-litre water bottles (80 mm diameter), one full and the other half-full, two ellipsoid shaped balls (62 mm maximum diameter), one soft and the other of stiff plastic. Reflective markers have been placed on the subjects’ end-effector to obtain, through a motion capture system, kinematics parameters of the movement. In the present article, the marker on the hand’s palm is considered for the end-effector movements. Further information on the dataset are in [An experimental dataset publicly available in  is used t are in .The subjects’ end-effector average trajectory is evaluated on 7 trial trajectories performed during the motor task, for each of the five objects. An example of the trial trajectories of the end-effector (dash-dotted lines) and their average (thick black line) is shown  for a giWobs and IDobs, respectively, with φ = 0.95 and for s* evaluated from zero  to the final rest position  highlights three phases, each of them characterized by a travelling action (gross movement) and a positioning action (fine movement). The first phase (reaching phase) consists of moving the end-effector  from the starting point to the grasping position; the second phase (transport phase) consists of moving the object from the grasping position and releasing it at the releasing position; finally, the third phase (return phase) consists of moving the empty hand from the releasing point to the starting point where the cycle ends . For each phase, the trend of Wobs is bell-shaped, with smaller values at points where a greater accuracy of the end-effector movements is required (positioning action). These results are in line with [Wobs values are higher in the task-irrelevant dimensions (travelling action of each phase), and reach local minima in the task-relevant dimensions (positioning action of each phase) where greater accuracy is required. The exploitation of motor redundancy is greater in the task-irrelevant dimensions since a greater motor variability is observed, witnessed by the large Wobs values. At points where accuracy is required, the environment forces the agents to reach a specific target. In short, each phase is characterized by a travelling action where the agent’s end effector is free to explore the space with greater spatial configurations, followed by the positioning action where accuracy is required, resulting in minima values of Wobs. At points where accuracy is required, the lowest values of Wobs cause steeper slopes of the IDobs trend resulting in a greater increase of the motor difficulty.The trend of ine with  since WoDifferent analyses have been performed to test the following two main hypotheses:The manipulation of different objects represents a change in the external constraints, affecting the agent-environment interaction, and influencing the motor difficulty experienced by the subjects during the entire manipulation task. To test the hypothesis, a one-way repeated measures ANOVA on In  it has b® software (version 26). Assumption prior to the one-way repeated measures ANOVA for the F = 4.361, p < 0.003). By applying the post hoc test with Bonferroni correction, the statistically significant difference of p < 0.005). Results are in To verify whether the five different objects determine a relevant difference on the overall experienced difficulty in the manipulation task (quantified by These results confirm the first hypothesis: the proposed stochastic Index of Difficulty is sensitive to capture the different motor difficulty experienced by the population while executing a motor task (object manipulation) with different external conditions (objects). In particular, subjects experience a greater motor difficulty when manipulating the bottles , compared to the ellipsoid balls (soft and plastic). Since results do not provide relevant differences between the two bottles , and between the two ellipsoid balls (soft vs. plastic) it can be concluded that the different mass alone as well as, the different stiffness of objets do not influence the population’s motor difficulty; the same conclusion holds in  where itF = 4.010, p < 0.02). By applying the post hoc test with Bonferroni correction (α = 0.05/3 = 0.016), the statistically significant difference of p < 0.001). Results are in A further investigation can be carried out focusing only on objects characterized by different geometry . The following analysis is aimed at verifying if the different geometry is the main object’s feature that affects the subjects’ motor difficulty. Assumption prior to the one-way repeated measures ANOVA for the These results do not support the hypothesis that the geometry is the main feature that affects the subjects’ motor difficulty. Nevertheless, statistical differences can be associated with the grasp type of the objects: the tennis ball  and ellipsoid ball (ellipsoid) are gripped with the same type of grip (’power sphere’ grip), while the bottle (cylinder) is gripped with the ‘large diameter’ grip , 59. No In this section, it has been shown that the manipulation of different objects affects the stochastic Index of Difficulty, and in particular, agents experience a different motor difficulty when manipulating objects with different grasp types. In the next section, further analysis has been performed to investigate how objects characterized by a given grasp type, affect the speed of execution and the stochastic width.The velocity profiles related to the manipulation of the five objects are bell-shaped . FeatureDifferences in the velocity profiles are accentuated halfway the path of each phase, where the peak velocity is reached; these differences are not substantial during the return phase since subjects do not handle any object. Furthermore, in both the reaching and transport phases, it can be observed that the velocity profiles of the two bottles  and of the two ellipsoid balls (soft and plastic) are almost overlapping, confirming that neither the mass nor the stiffness influence the speed of execution .F = 39.234, p < 0.0001) and transport phase  = 45.219, p < 0.0001). By applying the post hoc test with Bonferroni correction, the statistically significant difference of velocity peaks is evident in case of the ellipsoid/tennis ball vs. bottle (p < 0.001) in both the reaching and transport phases , results show statistically significant differences in the reaching (t phases .These results confirm that velocity peaks (and velocity profiles) of movements executed to reach, grasp, move and release an object are affected by the object-related grasp type. Again, the geometry of the object is not the only feature that influences movements performed, but also the volume. These two features together characterize the object’s grasp type , 59. MorWobs for each given object (averaged over 31 subjects) is bell-shaped in each phase of the task, with smaller values where accuracy of the end-effector’s position is required  are consistent with the velocity ones: events that require greatest accuracy  values) occur at the lowest speed values, and vice-versa. Furthermore, the maxima of both stochastic width and velocity profiles increase as the distance travelled within each phase increases . A greater travelling distance allows the subjects to go faster (The bell-shaped profiles of o faster  and attao faster .IDobs) by considering the stochastic behaviour of agents performing a given repetitive motor task. The agent’s motor difficulty is no longer a deterministic function of task geometry, but it is indicative of the stochastic behaviour of an agent as it performs a motor task while interacting with the environment: the motor difficulty is not a unique characteristic of an agent, but it is the unique characteristic of the combined agent-environment system.Starting from the original definition of the Index of Difficulty given by Fitts , the papBy applying the stochastic Index of Difficulty to the three-dimensional manipulation of different objects, a higher motor difficulty has been experienced by subjects when handling objects characterized by a “large diameter” grip (bottle), compared to objects characterized by a “power sphere” grip ; the different mass and stiffness of the considered objects, instead, did not influence the subject’s motor difficulty. Results obtained can help future work in which a complete factorial experimental analysis will be designed to investigate the effects of multiple objects’ features , and their interaction; the collection of a large dataset will be necessary, where the objects manipulated can be compared within and across multiple features, and agents with different motor abilities  will be recruited. Nonetheless, the current dataset and analysis has provided useful information on what could be, among many, the main relevant factors in object manipulation tasks, hence the opportunity to formulate more refined hypotheses in the future about the effect of specific objects’ features on the motor difficulty.IDobs can be used to design the layout of the workplace, such as in assembly workstations, since the kinematics of operators and motor strategies to be adopted to reach points are affected by the presence of obstacles, positions of targets, positions of components/parts to handle etc… [IDobs can be a valuable tool to be employed in job rotation/resource allocation problems. As an example, by observing a reference operator that executes optimally a given motor task, its motor difficulty can be quantified; this information can be employed to compare the IDobs of other operators and choose the operators whose motor behaviour it the closest to the ‘optimal’ one.The use of the stochastic Index of Difficulty, and the evaluation of the agent’s motor difficulty, is not limited to the specific application of manipulation tasks but can be employed in multiple fields of investigation. As an example, dle etc… , 64. FurFurther research will be focused in applying in industrial contexts the analytical-stochastic model proposed to analyse the motor difficulty of differently skilled operators in executing manual operations characterized by different motor complexity. Analogous research approaches can be translated to the context of motor rehabilitation.


In recent years, electroencephalograph (EEG) studies on speech comprehension have been extended from a controlled paradigm to a natural paradigm. Under the hypothesis that the brain can be approximated as a linear time-invariant system, the neural response to natural speech has been investigated extensively using temporal response functions (TRFs). However, most studies have modeled TRFs in the electrode space, which is a mixture of brain sources and thus cannot fully reveal the functional mechanism underlying speech comprehension. In this paper, we propose methods for investigating the brain networks of natural speech comprehension using TRFs on the basis of EEG source reconstruction. We first propose a functional hyper-alignment method with an additive average method to reduce EEG noise. Then, we reconstruct neural sources within the brain based on the EEG signals to estimate TRFs from speech stimuli to source areas, and then investigate the brain networks in the neural source space on the basis of the community detection method. To evaluate TRF-based brain networks, EEG data were recorded in story listening tasks with normal speech and time-reversed speech. To obtain reliable structures of brain networks, we detected TRF-based communities from multiple scales. As a result, the proposed functional hyper-alignment method could effectively reduce the noise caused by individual settings in an EEG experiment and thus improve the accuracy of source reconstruction. The detected brain networks for normal speech comprehension were clearly distinctive from those for non-semantically driven (time-reversed speech) audio processing. Our result indicates that the proposed source TRFs can reflect the cognitive processing of spoken language and that the multi-scale community detection method is powerful for investigating brain networks. For example, in an experiment designed to study the N400 amplitude and word expectancy, it was found that N400s to sensible and equally low-cloze-probability completions of strongly constraining sentences  were much larger than those to high-cloze-probability endings (The bill was due at the end of the month)  cannot be used to perform denoising properly, where I is the subject number. For this reason, we propose using a functional hyper-alignment method for eliminating this effect. The main idea of the functional hyper-alignment is to rotate xi and xj  to maximize their correlation among subjects. So far, several methods have been proposed for this purpose, such as group task-related component analysis (gTRCA)  is used to linearly map the speech envelope and the neural response in the sources by approximating the brain as an LTI system , the neural response signal ROIn can be described as follows.To evaluate the brain functions explicitly, we calculate the TRFs from speech input to sources in the brain cortex, instead of the electrodes on the scalp. The mTRF toolbox (s(t) are obtained from a gammatone filterbank followed by a power law  can be decoded from the source neural signal Under the theory for the LTI system, the backward approach can be modeled using a decoder The encoder The brain network can be characterized as a community structure. Therefore, community detection is often used in exploring the brain network during a given task  and H(Y) are the entropy for X and Y, respectively. I is the mutual information between X and Y.where When VI is equal to zero, it represents the most stable partition across densities  were recruited from Tianjin University and Tianjin University of Finance and Economics. The experiments were conducted in accordance with the Declaration of Helsinki  with a complete storyline, recorded by a male Chinese announcer in a soundproof room. The other 24 trials were the same story segments but were time reversed. All stimuli were mono speech with a 44.1 kHz sampling rate, and the stimulus amplitudes were normalized to have the same root mean square (RMS) intensity. The 48 trials were randomly presented to the subjects. All speech segments were also modified to truncate silence gaps to <0.5 s  at a suitable volume (around 65 dB). During each trial, subjects were instructed to focus on a crosshair mark in the center of the screen to minimize head movements and other bodily movements. There was a 5 s interval between each trial, and the subjects were given a 5 min break every 10 trials. After each story trial, subjects were asked immediately to answer multiple-choice questions about the content of the story to ensure that they focused on the auditory task. We embedded unique tones in some trials to draw more of the subjects' attention to the reversed stimuli. Subjects were requested to detect the tones and indicate how many times they appeared after the trial. The EEG data corresponding to the embedded tones was removed in further analysis.https://sccn.ucsd.edu/eeglab/index.php) in MATLAB (MathWorks)  at a sampling rate of 1,000 Hz. Six of the channels were used for recording a vertical electrooculogram (VEOG), a horizontal electrooculogram (HEOG), and two mastoid signals. The impedance of each electrode was kept below 5 kΩ during data acquisition. Three subjects' data were discarded in further analysis because they did not give a proper answer to the multiple-choice questions or the electrodes detached during the EEG data recording. The raw EEG data were pre-processed using the EEGLAB toolbox . The results of the ANOVA demonstrated that the prediction accuracy of our proposed method, the functional hyper-alignment method combined with the additive average method, was the highest among the three methods. A post-hoc test for the ANOVA showed that the prediction accuracy was significantly improved by the functional hyper-alignment method, compared with the average without the functional hyper-alignment method  and single-trial method . Therefore, the functional hyper-alignment with the additive average method was used in the following TRF estimation.The accuracy of the backward prediction of stimulus input from neural output is usually used for evaluating the performance of TRFs  . In t-test, p = 2.1 × 10−14; effect size d = 2).To address the differences in the brain networks between the natural speech and time-reversed speech in detail, we investigated the difference in the estimated TRFs between the two cases. According to Beres , the difThe investigation above was carried out with a brain network with full connection. As we knew, different speech functions have different brain network communities as speech planning, semantic processing networks. Therefore, some edges in the connection matrix may be invalid or noise for a specific speech process. It is thus necessary to determine the optimal connection matrix on the basis of the different thresholds of the connection.As described in the section on our method, we analyze the functional connection matrix using different densities, and we investigate the functional brain network at multiple scales , prefrontal cortex, and temporal pole are the main brain areas of the default mode network (DMN), which accumulates and integrates information over hundreds of seconds with our intrinsic information of memories during story perception , which is related to effortful comprehension and phonological decoding, and the supramarginal gyrus and parahippocampal gyrus, which are related to sentence-level processing. The parietal cortex is considered to be involved in both auditory and visual sentence comprehension. Additionally, the banks of the superior temporal sulcus and postcentral gyrus are important in complex sentence-level processing  audio processing. The functional brain network can be explained on the basis of the achievements of past research. It was also verified that the multi-scale community detection method is suitable for source reconstruction-based brain network studies.The datasets generated for this study are available on request to the corresponding author.The studies involving human participants were reviewed and approved by Research Ethics Committee of Tianjin University. The patients/participants provided their written informed consent to participate in this study.DZ performed the experiments, analyzed the data, and prepared the manuscript. GZ inspired the research idea and revised the manuscript. JD interpreted the results of the experiments and revised the manuscript. MU provided the code for extracting the speech feature based on cochlear filterbank and revised the manuscript. XL provided the idea and programmed the code for community detection. All authors approved the final version of the manuscript for submission.This research was supported by the National Natural Science Foundation of China (No.61876126), a Grant-in-Aid for Scientific Fund for the Promotion of Joint International Research , and in part by JSPS KAKENHI Grant (20K11883).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


With the wide application of Internet of things (IoT) devices in enterprises, the traditional boundary defense mechanisms are difficult to satisfy the demands of the insider threats detection. IoT insider threat detection can be more challenging, since internal employees are born with the ability to escape the deployed information security mechanism, such as firewalls and endpoint protection. In order to detect internal attacks more accurately, we can analyze users’ web browsing behaviors to identify abnormal users. The existing web browsing behavior anomaly detection methods ignore the dynamic change of the web browsing behavior of the target user and the behavior consistency of the target user in its peer group, which results in a complex modeling process, low system efficiency and low detection accuracy. Therefore, the paper respectively proposes the individual user behavior model and the peer-group behavior model to characterize the abnormal dynamic change of user browsing behavior and compare the mutual behavioral inconsistency among one peer-group. Furthermore, the fusion model is presented for insider threat detection which simultaneously considers individual behavioral abnormal dynamic changes and mutual behavioral dynamic inconsistency from peers. The experimental results show that the proposed fusion model can accurately detect insider threat based on the abnormal user web browsing behaviors in the enterprise networks. With the significant growth of Internet of things (IoT) devices, the IoT system has been further developed and extended . The intThe heterogeneity and diversity of IoT devices make the IoT face greater security threats ,5,6,7. TThere are many IoT devices around us, that can sense, store, compute, and communicate information, and insiders can easily commit data leakage, malware, and Denial-of-Service (DoS) attacks through IoT devices . The ideWhat’s more, web traffic features are extracted and input into the classification algorithms, such as decision tree and random forest, to detect malicious web behaviors. However, the effectiveness relies on enough labeled samples which are difficult to collect due to the lack of malicious samples of the IoT insiders. In addition, web browsing behaviors have been studied for the e-business workflow and recommendation system, however these methods analyze the web browsing patterns within a specific server and cannot be applied directly in the IoT insider threat detection.In this paper we firstly present the individual user behavior (IUAD) model that characterizes the dynamic change of user behavior based on power distribution to detect the deviation with respect to the usual normal behavioral change pattern of each user, as observed in a prolonged period of time, and then we propose the peer-group behavior (PGAD) model to evaluate the mutual behavioral similarity among users in one peer-group under the assumption that users in one peer group usually have common jobs and exhibit consistency of behavioral dynamic change. More specifically, to automatically identify the peer-group, the relationship network of users is built based on the co-occurrence of browsing behavior and then the peer groups are extracted from the relationship network of users using the dominant set framework. Furthermore, we develop the fusion model that combines IUAD model and PGAD model to detect insider threat from the perspectives of personal dynamic browsing anomaly and dynamic inconsistency among one peer group.In summary, the proposed method makes the following main contributions.We propose an individual user behavior model (IUAD) to characterize the change of user browsing behavior based on power distribution under the assumption that the rate of individual user behavior change is continuous and user-specific.We put forward the peer-group behavior model (PGAD) model that automatically identifies the peer-group by applying the dominant set method to relationship network of users, and compare the mutual behavioral consistency among all users in the peer-group under the assumption that the user behavior is similar with his/her peers.We propose a fusion model for the insider threat detection that identifies the deviation of the individual dynamic behavior from two complementary perspectives: dynamic change of individual user behaviors and the dynamic inconsistency of user behaviors among all users in the peer-group.There are studies in different domains analyzing web browsing behaviors. Erdem et al. proposed an e-business workflow that is designed to generate test scripts based on user browsing behaviors. The e-business workflow is able to generate browsing patterns that are unseen in the historical web usage data . BhuvaneIn addition, there are many studies detecting malicious web behaviors based on web traffic analysis. Al-Bataineh et al. proposed a classification algorithm to identify malicious data stealing attempts within web traffic. The classifier uses entropy and byte frequency distribution of HTTP POST request contents as features . Ranjan N domains from the target computer are chosen to form the profile vectors of the target computer. The candidate computer with the highest cosine similarity for the target computer has the highest probability that it is used by the same user. On the other hand, support-based and lift-based user profile models pick the top patterns from each user and union them to form the set of candidate patterns [A few works have been proposed to build user profile from user’s web browsing behavior for the purpose of user identification. From the perspective of page access frequency and page view time, four web user profile models , called patterns . Then thWe propose an insider threat detection system that builds a fusion model from user web browsing behavior. The overall structure of the proposed model is illustrated in U be the set of users and R be the set of domain names accessed by users. And, let Q be the set of browsing records, such that n different time periods based on fixed time span c, each time period consists of a few records, i.e., n time periods for one user, the first k time periods To formalize the problem studied in this work, we will use the following notation. Let tth time period b are parameters to learn by employing least-squares approximation.To study the behavioral change pattern of the individual user, we collect web browsing records from 4 users in our lab over the months of January to May 2018. The domain names visited in chronological order are presented in tth time period. We focus on the deviation when the actual sequence number is larger than the predicted one, and the larger the deviation the more anomaly the user behaviors. Because the insider’s malicious activities usually require accessing new resources that he has not visited before, we consider it normal if the actual sequence number is approximately equal to or less than the predicted one.Since the normal pattern of continuously evolving domain names is supposed to be the power distribution and is formalized in the Equation , the abnFinally, the abnormal scores of different test time periods for one user derived from the IUAD submodel can be represented as a vector: Since users are not isolated in the insider system. One user’s behavior should be in line with his/her peers. In an insider threat detection context, the intuition is that user activity should reflect the user’s job role and users with similar job role should exhibit similar behavior in web browsing behavior. In this section, due to behavioral similarity among users in one peer-group, first we introduce the relationship network of users, second peer-groups are extracted from the network and we detect abnormal web user behaviors that are unusual compared to the user’s peers in the same peer-group. Instead of analyzing users independently in IUAD, PGAD analyzes user browsing behaviors that are not common to observe over the entire peer-group.The relationship network of users is constructed as a completely connected undirected graph. The example procedure is illustrated in B of size In order to obtain the distance between users, the users and visited domain names firstly are mapped onto a bipartite graph, such that users and domain names are modeled as vertices, and an edge represents the number of times that a user accessed the domain name. We summarize the information in this bipartite graph of users and domain names in an adjacency matrix k periods, the distance between two users can be represented by a corresponding Based on the adjacency matrix computed from the records in the first And then the distance is transformed into similarity (edge-weight) using a Gaussian kernel and the symmetric similarity matrix Equation  is rewriTo automatically discover the peer-group, we adopt the dominant set technique  to iteraNext, the overall similarity between user A non-empty subset of users On account of the behavioral similarity among users in the peer-group, the abnormal user behavior can be measured based on the user’s behavioral deviation from his peers’. Correspondingly, we score the user Equation  with theFinally, the abnormal scores of different test time periods for one user derived from the PGAD sub-model can be represented as a vector: In this section, we propose the fusion method based on IUAD and PGAD to comprehensively consider the behavioral change of individual user and behavioral consistency among users in the peer-group for abnormal behavioral detection in enterprise networks.One of the issues for fusion is that the scores are generated from different mechanisms and as a result, provide no common ground for comparison. To counter this, the scores firstly have to be normalized.w = 0, only IUAD is used. Consequently, the finally abnormal scores of different test time periods for one user from the fusion method can be represented as a vector: Subsequently, we formulate the fusion scheme as a weighted linear combination of the IUAD and PGAD:Due to the lack of availability of proper web browsing logs in enterprise network, we utilize an insider threat dataset published by CERT Carnegie Mellon University for this research. The 17-month period dataset “R4.2.tar.bz” consists of five types of records of 1000 employees, i.e., HTTP, File, device, logon, email records. All HTTP records are chosen for this analysis. Each record contains user, PC, URL and web page content with time stamps. The web page content is ignored and an HTTP record is depicted in First of all, we divide all HTTP records into a number of samples and every sample is composed of records of one day. And then we split these samples into two different sets: construction and validation. The construction set is composed of a certain percentage of the samples and is used to create and train user behavior model. The validation set consists of the remaining percentage of the samples and the full set of user browsing behavior anomalies, and is used to yield a detection performance. Specifically, for each experiment, we split the dataset with different percentages for both sets, construction and validation, namely: 80–20, 70–30, 50–50, 30–70, and 20–80, to make a sufficient comparison.To emit an evaluation, we compare the abnormal scores of the samples in the validation set against a threshold. A sample is labeled as anomaly, if its score is higher than the threshold, and normal, otherwise. We vary the threshold from 0 to 1 to study the performance of our model, thereby drawing a so-called Receiver Operating Characteristic (ROC) curve. So, we start with a low threshold, getting a lax model; then, we increase the threshold slowly until we get a very strict one. Doing so, we have got results from 100% False Positive Rate (FPR) with 0% False Negative Rate (FNR), to 0% FPR with 100% FNR.In order to compare detection models one another, we have used two different measurements: Area-Under-the-Curve (AUC) and the Minimum Misclassification Point (MMP). AUC denotes the area under a ROC curve. An AUC equal to one amounts to the perfect model, which correctly marks every sample, as normal or abnormal. Conversely, an AUC equal to zero corresponds to the worst model ever. What’s more, MMP denotes a point in the ROC curve that minimizes FPR+FNR. The smaller the MMP, the better the detection model.In order to process the large experimental dataset and get the experimental results quickly and effectively, we choose a Linux server Dell R740 as the experimental platform and the experimental simulation environment in this paper was follows: Ubuntu 20.04 64-bit, 125 G RAM, Python 3.7. The configuration of the server can provide an ideal environment for the running of the experimental Python programs.w, which is the trade-off term for combining IUAD and PGAD sub-models. The result is shown in In this section we evaluate the performance of our fusion model in detecting browsing behavior anomaly. We vary the hyper-parameters w between 0.1 and 0.3, at which both sub-models are combined most appropriately. Besides, we can notice that as we shorten the amount of available training, the fusion method is most stable compared to the two sub-models from the perspective of standard deviation of the MMP and achieves the best AUC value for the 70–30 (%) experiment.Looking closely at the table, we conclude that the fusion model outperforms the two single sub-models IUAD and PGAD, since the central part of w is set to 0.1 and 0.3 whenever the amount of available training in different graphs.We can clearly appreciate in Summarizing, the fusion method achieves better performance than IUAD and PGAD that validates the effectiveness of our fusion scheme and the complementary capacity of two sub-models. Based on the combining factor between IUAD and PGAD, the fusion method can capture user browsing behavior from both the dynamic change of individual user and behavioral similarity among users in the same peer-group.For comparison purposes, we have implemented and tested TF-PVN , TFIDF-PN domains from the target computer to form the profile vectors of the target computer and then apply the cosine similarity measure to assess the similarity of two browsing histories. In our experiment, we select the top one-sixth of the domains as suggested in the TF-PVN and TFIDF-PVN and employ the cosine similarity measure to assess the anomaly of the user current behaviors compared to the past activity. While, support-based and lift-based user profile pick the top patterns from each user and union them to form the set of candidate patterns. The distance between two profiles is computed based on Euclidean distance. In our experiment, we choose top 10 domains from each user as suggested in the support-based and lift-based user profile method and use the Euclidean distance to assess the abnormal score of the user’s current behaviors based on the history browsing records.TF-PVN and TFIDF-PVN choose top Due to the best performance, we use With the growth of IoT, new security challenges arise in the existing security frameworks. Because internal staff can easily access the network environment through IoT devices, it is a great challenge to detect internal attacks. In the paper, an insider threat detection model is introduced, which models user web browsing behavior. At the same time, the model is a comprehensive model, which detects the deviation from the power distribution of individual user browsing behavior and behavioral similarity among users in the peer-group to the user’s current behavior simultaneously. To automatically identify peer-group in an organization, relationship network of users is constructed based on an adjacency matrix that summarizes the frequency with which a user accesses a domain name in the history, and then the dominant set technique is applied into the relationship network of users to uncover a group of users with similar browsing behavior as one peer-group. After a wide range of experiments, it is verified that the proposed fusion model obtains best performance when we set the trade-off actor for combining IUAD and PGAD sub-models as


With the rapid concurrent advance of artificial intelligence (AI) and Internet of Things (IoT) technology, manufacturing environments are being upgraded or equipped with a smart and connected infrastructure that empowers workers and supervisors to optimize manufacturing workflow and processes for improved energy efficiency, equipment reliability, quality, safety, and productivity. This challenges capital cost and complexity for many small and medium-sized manufacturers (SMMs) who heavily rely on people to supervise manufacturing processes and facilities. This research aims to create an affordable, scalable, accessible, and portable (ASAP) solution to automate the supervision of manufacturing processes. The proposed approach seeks to reduce the cost and complexity of smart manufacturing deployment for SMMs through the deployment of consumer-grade electronics and a novel AI development methodology. The proposed system, AI-assisted Machine Supervision (AIMS), provides SMMs with two major subsystems: direct machine monitoring (DMM) and human-machine interaction monitoring (HIM). The AIMS system was evaluated and validated with a case study in 3D printing through the affordable AI accelerator solution of the vision processing unit (VPU). The incredible pace of deployment of AI technology is driving the sudden emergence of the fourth industrial revolution (Industry 4.0). The rapid evolution of this manufacturing paradigm is driving an urgent need to create useful tools for designing and deploying diverse data models. Small and medium-sized manufacturers (SMMs) typically lack the appropriate resources to affordably establish the IT infrastructure needed for gaining visibility of factory operations  and real-time machine monitoring which is the promise of Industry 4.0. Using consumer visualization tools developed for smartphones, smart home applications, and utility smart meters installed by utilities will ease information technology (IT) infrastructures’ demands for acquiring data to assess workflow and energy/materials usage. By classifying workers’ activities and traffic trajectory inside the facility, the states of equipment usage and the workflow can be acquired. A system identifying the correlation between the states of equipment/workflow enables workers and supervisors to gain insight into the operational footprint. The continuous streaming machine status and manufacturing operation insight can optimise factory operation for production efficiency and dynamic scheduling of equipment used to lower energy costs in multi-tier electricity supply markets. This coordinated data acquisition system from visualization and state sensors is a critical IT infrastructure for smart manufacturing.The coordinate data acquisition system is especially crucial for the affordable, scalable, accessible, and portable (ASAP) solution of smart connected worker advanced manufacturing systems. For the AI embedded in these systems to generalize to a broad context, these systems will need to be as diverse in their architectures as the manufacturing ecosystems they will operate. These manufacturing ecosystems are computationally bound through a network of manufacturing activities, tools, machinery, and inputs  configured through data-enabled manufacturing processes and enacted by workers augmented with additional intelligence to perform adaptive manufacturing tasks. The challenge is to develop design tools that will enable engineers to reliably assess and implement data flow requirements for ASAP smart connected workers across all manufacturing sectors while focusing on computer vision-based approaches for improving productivity, automated demand-side management, detecting errors, minimizing waste streams, and improving sustainability performance.To use reliable methods to automate these supervisory tasks, this research proposes an Artificial Intelligence-assisted Machine Supervision (AIMS) system, which tracks the working state of the machines through real-time observation of the machine. The AIMS does not rely on any data transmitted directly from the machine to determine the machine’s status. If the machine’s control system fails to detect an abnormal condition, the AIMS can detect the abnormality by observing the machine. When building an AIMS on a particular machine, the designer must thoroughly understand the machine’s possible standard workflow in the context of human interactions. In practice, the AIMS will immediately detect anomalies when the machine components go beyond the range specified by the standard workflow.Recent works have proposed to advance the processes of industrial production and supervision. Bian et al. proposed and developed a centralized and automated real-time monitoring system, utilizing numerous machine learning-based techniques to reduce the cost of human labor and improve energy efficiency in smart manufacturing . Peres eObject detection has seen increasing application in various fields, most recently in smart manufacturing. Wiech et al. applied an optical object detection system in a learning factory setting to detect human errors [n errors . Overcomn errors . Lemos en errors . Leveragn errors .Text recognition is also rapidly developing due to the advancement of various AI techniques. Wang et al. addressed some challenges of text detection in the industry through an algorithm called Progressive Scale Expansion Network, which resulted in high-performance accuracy [accuracy . Yu et aaccuracy . Based oaccuracy . Hu et aaccuracy .A novel method for anomaly detection and state change detection based on the AI-assisted method;To significantly reduce the computational difficulties while clarifying logic and reducing system maintenance difficulties;Deployed cloud computing technology and developed the system on embedded devices to make it affordable for SMMs to implement the AI-based Smart Manufacturing practices;Incorporated the human-computer interaction monitoring to ensure that the corresponding action has an expected impact on the machine state, which has safety and cybersecurity implications.The developed AIMS system follows the same logic as traditional supervision methods and can improve the effectiveness of human supervisors in manufacturing scenarios, thus improving productivity and reducing labor costs. Specifically, we summarize the contribution of this work as the following items:The proposed AIMS system consists of two major subsystems: direct machine monitoring (DMM), and human-machine interaction monitoring (HIM), as shown in The AIMS applies to any manufacturing environment but is particularly relevant in scenarios with manual operations or facilities with fewer automation features that need real-time monitoring or supervision  In actual production, it is required to supervise the machine’s working state at all times. When a specific machine is identified for the application of the AIMS, an analysis of how the machine works are required before designing an AI-supervised model to achieve more accurate and efficient supervision. The following analysis should be performed for a specific supervised device in the AIMS pipeline.For most manufacturing equipment, it is not easy to understand the operating conditions from direct observation of the entire machine. AIMS utilizes the real-time monitoring of the motion of the major components  inside the machine to analyze the operating conditions of the entire machine. While this paper is explicitly focused on motion in the sense of state changes in physical space, the approach can also be applied to a more generalized concept of motion which can be extended to radiating systems like heaters and radios, as discussed in future work. This component-based identification provides a more precise observation accuracy than direct observation of the entire machine. It is also an accurate way to analyze the machine’s status based on the internal structure. The following section explains how the implementation of this approach uses artificial intelligence-based objection detection.Building the AIMS for a particular class of machines requires the definition of the state space, which includes the operating states of the overall machine, operating states of the major supervised components, and the constraints associated with prescribed operational parameters  The machine state transition diagram graphically represents operational states and these associated constraints. Individual machines within a particular class will likely exhibit some variation from the class-based state diagram addressed in AIMS through the implementation of adaptive AI training. After confirming the current state, the constraints on the primary component corresponding to the current state are analyzed. Once our supervised model detects broken constraints, the system can recognize a machine’s operational envelope problem. At the same time, AIMS makes a state diagram of the transition relationships of each state. The supervised model can obtain a set of possible subsequent states from the state diagram based on the current state. For each possible subsequent state, the model can check the motion state of each major component to confirm whether the threshold to enter the state is satisfied. In our experiments, a boolean First-in First-out queue of capacity five is applied to record the thresholds of the last five frames for each possible next state. If 80% of the thresholds in the last five frames are satisfied, the machine is determined to enter the next state corresponding to the queue. The use of state diagrams dramatically reduces computational complexity while improving accuracy. Because the AIMS model understands the state diagram, at any point in time, the model can call the validation function to determine whether each major component is in a legitimate state based on the state diagram instead of calling a tedious general error checker, which can be error-prone and computationally prohibitive. Moreover, the use of state diagrams also dramatically reduces the computational effort of state transformation. Without state diagrams, a model would need to analyze the motion states of the major component of multiple consecutive frames in the past and analyze each possible combination of future states. Such an analysis is highly complex. With the help of state diagrams, inputting the current state helps the model invoke the exact threshold that needs to be checked. This reduces the amount of computation, streamlines the logic, and significantly reduces the possibility of incorrect codes and the difficulty of maintaining the code.For the system to have more accurate predictions, the working environment of the manufacturing equipment also needs to be considered. Of particular note are those conditions that can affect the computer vision model. With the understanding of the working condition, a more robust model can be trained.L. The length of L is the number of objects recognized in the image. i-th element of the list, corresponds to the i-th object identified from i-th object and its position in pixel coordinates on the image. If there are n major components that AIMS needs to track, then the type is an integer in L is describing.The Direct Machine Monitor (DMM) utilizes computer vision to supervise the state of motion of the main components of a manufacturing machine. AIMS uses a general camera to capture images of the interior of the supervised machine in real-time. Each image L is generated describing the location states of the major components inside the machine at the current moment, after which the DMM retrieves the set S of possible states corresponding to the current state T of the machine during the process. Then for all L satisfies the transition threshold of T to state For any moment in the machine’s operation, the camera inside the machine captures a picture of its interior To get a more accurate assessment of the working state of the machine, AIMS uses another model to detect the human-machine interaction. This system observes the machine’s human operation from outside and calls the corresponding operation’s handler function to join the machine state’s logic judgment. Specifically, a complete human-machine interaction should involve the following exchange of information: (1) the human worker, with the knowledge of the working scenario, actively provides input to the machine; (2) the machine, upon receiving the input instructions, proceeds to work on its dedicated duty; (3) after finishing its job, the machine returns output to the worker, notifying its availability; (4) the worker, knowing that the machine is once again available, provides it with further instructions. Based on this logic, the proposed module is developed to capture and understand the human-machine interactions of an operating manufacturing system by analyzing the exchange of input and output information. This system is critical since it utilizes powerful machine learning algorithms to enable a flexible and easily deployable solution to monitor and analyze human-machine interactions without heavily relying on data communication protocols such as Tera Term and MTConnect.L, the DMM and the current machine state generate the operating state information.In typical scenarios, workers deliver input to the machines by performing operations on an interactive terminal. Such an operation may include pressing a button on the panel, entering commands via keyboard, etc. Similarly, the feedback of a machine is usually presented to the worker via text (or symbolic) format on the screen of the same panel. The proposed module consists of a wearable camera on the worker that captures the worker’s vision during a valid human-machine interaction to capture this kind of information flowing between the worker and the machine. The camera captures an RGB image Because the computer processes each frame much faster than the worker can operate, most frames’ recognized instruction status in Generally speaking, object detection relies on two different hardware systems: one is based on the generation of 3D point cloud information captured by LiDAR , which hThis model aims to monitor the machine’s state in real-time and detect machine failures in time. Faults include machine parts moving in a manner that does not conform to the current state, machine state transitions that do not conform to a predefined process, and machine states that do not conform to interaction expectations after worker interaction through the panel. Therefore, we defined the following criteria to measure the model’s performance.precision reflects the model’s ability to detect machine errors, and recall reflects the model’s ability to track the normal working state of the machine.After each frame is processed, the system compares the machine prediction state with the actual state. If they agree, the prediction is judged to be correct. Moreover, if the actual state is not In this section, we examine the feasibility and efficiency of the developed AIMS System in the use case of 3D printing. In this case, the general machine is the 3D printer.canonical right-handed coordinate system is considered, as illustrated in The work of the 3D printer relies on the synergy of three components: the extruder, the build plate, and the motor axis. The extruder is filled with raw printing material. At the bottom of the extruder is the nozzle that constructs the printed item. The The printer was placed in a simulated factory environment in our lab. Under these conditions, the main thing that affects the computer vision model is the surrounding lighting conditions. The required conditions in the lab are relatively simple. There is one central lighting system inside the lab and one inside the printer. These two lighting systems may be on or off while the printer operates. Therefore, when training the model, the four possible lighting conditions are considered to ensure that our model makes accurate predictions under each of these conditions. The lighting conditions are also included as a factor during testing.H by W tensor to represent the colours of each pixel of the image. The image contains H times W pixel for each channel and can be regarded as an H by W matrix. The most common and intuitive way to represent an object in pixelated images is to use a bounding box, which can be understood as a rectangle within which the object is placed. Mathematically the coordinates of the center of the rectangle on the H by W matrix can be used to determine the unique corresponding bounding box. After enough images are collected, the bounding boxes of the extruder, build plate, and motor axis in each image with the corresponding type of bounding box are labelled, which is the final step in data preparation for model training.The technique of artificial intelligence computer vision is leveraged in AIMS to automate the monitoring of the extruder, the build plate, and the motor axis. The AIMS AI computer vision system was configured to monitor the 3D printing system with main components consisting of the extruder, the build plate, and the motor axis to demonstrate model building. A camera was placed inside the 3D printer, ensuring the three main components were captured. The printer performed multiple print jobs with the camera to capture the images inside the printer at a rate of five frames per second. Each image is coded as a 3 (RGB channels) by The state-of-the-art methods for object detection are deep learning models based on convolutional neural networks (CNN), such as Region-based Convolutional Neural Networks (R-CNNs) , and SinAlso of interest are Single Shot Detectors (SSD) which consist of a backbone neural network for extracting features and convolutional layers for making predictions. Unlike R-CNN, SSD predicts both the classifications of objects and the bounding box locations simultaneously at convolutional filtering kernels. More specifically, input images fed to the SSD require predetermined ground truths bounding boxes and class labels before training. Subsequently, the image is separated into regions containing anchor bounding boxes predefined in shape and serve as initial guesses. The input image is encoded and passed through a pre-trained feature extractor, such as the Visual Geometry Group from Oxford (VGG16) , to obtaFor the construction of the direct monitoring model, several cutting-edge models were selected, trained, and compared in three dimensions to find the most suitable model to handle the internal conditions of the 3D printer. Specifically, YOLOv3 , YOLOv4 Based on the tests, the accuracy of YOLOv4 has a clear advantage over the standard COCO test. However, this advantage was not evident in our 3D printer dataset. Also, the research team observed that YOLOv3 has a higher processing speed in the 3D printer than the other models. To best accommodate the use case of the 3D printer, the convolutional computing-based model YOLOv3 is adapted. YOLOv3 uses up to 53 convolutional layers, reducing the computational speed but improving prediction accuracy. Furthermore, when used in the AIMS, the computational speed of YOLOv3 is sufficient to complete the system’s work requirements. YOLOv3 obtains the final prediction through a multi-layer feature extractor. The first layer obtains the basic feature after multiple convolution calculations and then combines it with the feature of the previous unsampled data into the next extractor layer. In the last layer, it goes through a similar process again. With this three-layer design, the model can obtain both the original data and the features extracted from intermediate calculations to obtain more accurate predictions. When using this model, the prediction of the bounding box is made from three scales: (1) the position of the bounding box, (2) the confidence of the object present at that position, (3) and the class of the object labeled by the bounding box.Preprocessing. Before feeding the raw images captured by the worker’s headset camera into the subsequent machine learning models, preprocessing is required to ignore unnecessary background details and to narrow down the regions of interest  to achieve optimal efficiency, as illustrated in Text Detection and Recognition. After locating the display screens of the machine, the module performs text detection and recognition on the filtered regions to extract the machine’s output, which is illustrated in Locating Push Buttons. To monitor the human’s input to the machine, the module first detects the locations of the push buttons that the human interacts with. Since the buttons may be of various shapes and sizes in diverse working scenarios, the proposed model predicts the position of the buttons via specific calculations rather than depending on the OpenCV’s functionalities. In this case study of the 3D printer, there is a push button to the left of each text display screen. Therefore, after identifying the contour coordinates of the display screens in the previous preprocessing steps, the module calculates the relative positions of the pushbuttons. Since the wearable camera is located on the headset of the human operator, the image frames captured will inevitably contain certain distortions and tilt. Therefore, the algorithm for locating the button regions should accommodate different perspectives of the human operator’s vision, as illustrated in Since the positions and sizes of the buttons relative to those of the text regions do not change when there is a transformation of the camera’s perspective and angle, the algorithm can identify the regions of the buttons by extrapolating the contour coordinates of the text regions following the perspective transformation, as illustrated in x-coordinate  and the y coordinate  of every corner vertex need to be calculated to determine and draw the bounding boxes of the button regions. The ratios of lengths are defined as:For each scenario, the Due to the properties of similar triangles, these ratios of lengths will not change dramatically during the perspective transformation.For the first scenario in which the camera angle is horizontal, no specific transformation is required since the corner coordinates of the button regions and the text regions are aligned horizontally and vertically. The length of the text region contour can be calculated asTherefore, the For the second scenario in which the camera angle is left-tilted, the coordinates of the button vertices are no longer at the same horizontal and vertical angle. Since there is a shift in vertices’ positions, the lengths and heights’ calculations need to be changed accordingly. The top lengths and heights of the text region contours are defined as: For the third scenario where the camera angle is right-tilted, a similar transformation of coordinates can be applied to obtain the relative positions of the button regions on the image frames. By feeding the Finger Detection. After locating the press-button regions, the interactions and inputs from the human operator can be captured by detecting whether the finger is present above certain buttons. Instead of adopting finger detection models implemented with computer vision techniques, the module utilizes OpenCV’s extract pixel color values from frames. Specifically, with the bounding box regions  calculated in the previous step, the module extracts the RGB color values for each pixel within the button regions. By calculating the averaged RGB values for each button, the module identifies which button the human operator interacts with by picking out the button position with the most distinct averaged RGB values. To deal with edge cases in which the human operator’s finger covered more than one button, the module always prioritizes the button position with the least x coordinate value , where the fingertip is assumed to be located at. In other possible cases, when the text-display regions are occluded (which happens most often when the right hand is used to operate the panel), the areas blocked for more than 50% area are omitted. Since in most human-machine interaction scenarios, the fingers other than the index finger are curled up, the occlusion of text regions by the hand and fist should not interfere with the button regions of interest that the index finger is positioned at. Therefore, these methods detect the left-hand and the mirrored right-hand interactions with the panel press buttons. Consequently, this module only focuses on the interactive button regions of interest. It can thus achieve greater accuracy than machine learning models that try to identify and predict the whole finger globally. Therefore, even though this implementation of finger detection is less sophisticated than those that adopt computer vision models, its simplicity and efficiency are most suitable for the real-time monitoring of human-machine interactions.Before the DMM training, the 3D printer printed two items. At the same time, a camera inside the machine recorded the entire working process. During this process, the experiments continuously switched between four possible lighting conditions. The final 1500 images were put into the training set, with about a quarter of the overall images from various working conditions. The research team then annotated each image with the extruder, build plate, and axis with the bounding box.The cloud computing approach was used for the training of the model. The labeled 1500 training images were put on the Colab server  providedThe model was then deployed to the low-cost solution VPU NVIDIA Jetson Nano  with 128When the model finishes processing an image, it generates a predicted bounding box that encloses some pixels on the image, and we use A higher i confident predictions are included, and n is the number of predicted bounding boxes. The calculated For a given on curve  can be cThe numerical results in the average precision calculated with a bilenet  should The human-machine interaction monitoring module test images are collected from the wearable camera on the operator’s headset and during the same process described in The test accuracy for the text detection model is optimal for the setting in which the exterior lighting is dark, and the wearable camera is not tilted. The high contrast in the lighting of this scenario allows better detection of the display screen contours. At the same time, the leveled perspective rules out further complications in calculating press-button contours. The text detection model can achieve competent accuracy and robustness in all the test environments set for the experiment.The finger detection model, utilizing the method of color comparison of bounding-box regions, depends on the accuracy of the previous model’s press-button detection. In the 1500 total test frames, 1348 frames accurately identified all the press-button regions compared with the ground truths, achieving an average accuracy of approximately 89.87%. For all the correctly identified press-button regions, the finger detection model achieved 100% perfect accuracy in detecting which press button the operator’s finger is interacting with. Therefore, the entire HIM module’s performance mainly depends on the text detection model’s accuracy.The AIMS system was deployed on the 3D printer Stratasys uPrint in the lab. All the information tracked by the AIMS was transmitted to the graphic user interface in real-time through the LAN, making it easy for the staff to monitor the working status of the machine remotely when needed. Two tests were conducted to evaluate the effectiveness of the AIMS system.The first one was the machine’s regular operation test. In this test, we used the 3D printer to create ten items under various lighting conditions. AIMS has outstanding performance in regular machine operation. The AIMS followed the whole machine operation process in all nine tasks, including five times when a worker operated the panel. Afterwards, the 3D printer was adjusted to its initial state and was tested against the failed items again. This time the AIMS completed the job accurately.Within all frames of these ten tasks, we found that the model gave incorrect predictions or failed to detect the object in rare cases, three frames to be specific. However, since the AIMS system uses a boolean First-in-First-out queue buffer of up to five frames to determine if there is a fault or state change, as long as more than 80% of the model predictions are accurate, the AIMS system will work accurately. For example, if the model predicts correctly in the previous frames, and suddenly the lighting condition has changed, or something briefly blocks the camera’s vision. This might result in the misprediction of two or three frames. However, the condition returns for the subsequent frames normal; thus, the prediction continues to work as expected. In this case, these few frames of misprediction, not exceeding five consecutive frames, will not be considered as a change in the state of the entire machine but as an accidental mistake and will not influence the prediction of the entire machine. As a result, these few prediction errors did not cause the failure of AIMS.The second test is the machine abnormal condition test. In this test, we also used the 3D printer to create ten items under various lighting conditions. However, on the way to the printer’s task, the research team intervened several times to make the printer jump out of the pre-defined standard workflow. This was used to test whether the AIMS system could detect abnormalities in the machine state. AIMS also performed very well in the machine anomaly monitoring test. 3D printers were successfully monitored for state anomalies within five frames after artificially created anomalies in 9 out of 10 jobs. However, in 1 task, the research team stopped the 3D printer at Calibration, a state of readiness before the actual print job began. The team named this test as halt test. In actual production, such a state would be equivalent to a machine whose workflow is blocked in a particular state for some reason, thus causing production to stall. However, since the machine components are stuck in their legitimate range of motion in their current state when this fault occurs, AIMS’s Abnormal Checker does not consider this a fault. In the next step, a reasonable solution is to include a heuristic time estimate for each machine state when defining its workflow. If AIMS finds a state severely overrun, it issues a warning.For each frame during the test, the prediction results of AIMS are evaluated based on the scheme defined in The data shows that the model performs well in tracking the regular operation and detecting the fault. By taking a closer look at the prediction results, the team finds that most false negatives and positives are mainly located around the machine’s state transition. A 5-frame buffer must be filled before AIMS issues a state transition when the machine changes its state.ASAP  during application.The proposed method is affordable as all the hardware used in the proposed method is cost-effective compared with existing solutions. The requirement of image quality of the algorithm is not high, ordinary camera sensor units can be used instead of the ones with high specs, and thus is affordable. Running the algorithm for detection also does not require high computational power or expensive commercial software. The proposed method can work without a certain protocol and thus avoid the expensive installation fees of such systems. For some machine models, there exist some protocols that can access the machine log and get the machine state. However, it is common for different manufacturers to use different protocols for their machines, and a factory might use various brands of machines. In this case, some commercial software or certain hardware might be needed to monitor all the machines of interest, and installing them might account for a sizable portion of the budget. In the proposed algorithm, this challenge could be solved as it is compatible with the existing portals as a bonus but does not require any protocol or the installation of additional hardware systems.Specifically, it is scalable as it could be expanded and scaled up if the factory already has other existing sensors or IoT devices such as thermal cameras and smart meters to extend its functionality or increase its accuracy. With more sensors, the information the system can collect will significantly increase. By editing the proposed algorithm to make it compatible with the additional information and properly using it, the system can be extended in terms of accuracy and functionality. For example, based on the model’s prediction, a virtual tutorial can be easily added as its functionality, which can give step-by-step instruction to workers that are new to the operation of machines and detect misoperation to reduce the risk of operation. This algorithm can also be used for machines other than the 3D printer with minimal code editing and potentially on any machine where a camera can capture the human-machine interaction process. The relevant work has been published, where the proposed method is scaled to the more convoluted real-time monitoring of operators’ interactions with an assembly line [The proposed method is bly line .accessible as the framework is automated and fully encapsulated. It can be used by workers without advanced knowledge of computer vision or understanding of the protocols. Other methods of monitoring the machine state might require the workers to get familiar with the protocols of machines. If a factory uses machines from different manufacturers, each worker has to learn multiple kinds of protocols and needs to be very familiar with the machine log from each machine to keep track of the machine’s state. The proposed method does not require such knowledge. As long as the model is trained, the prediction process can be done automatically, and the prediction result will be shared with the workers in a very trivial way. Even in cases of using various machines, only minimal changes to the model need to be made. Once the model has been modified and trained by engineers, no additional effort needs to be made to understand the usage of the proposed system on different machines.The proposed method is portable as the data is collected from a portable camera, does not require a very high resolution for the prediction, and is processed by the portable NVIDIA Jetson kit. The proposed method can work perfectly in portable conditions by using portable hardware and lightweight visual data, which is fast for wireless data transmission. All the hardware involved in the detection process can be portable. For example, the camera can be installed on a pair of goggles or a helmet, which connects to a microcomputer system such as a Raspberry Pi.The proposed method is The proposed algorithm has two major limitations. The algorithm is not super robust for handling extreme cases when consecutive frames following a distribution not seen in the training sample were captured by the camera. All the existing machine learning-based object detection algorithms assume that the images will be processed following the same distribution as the training data. When the above assumption fails, an unexpected result may be generated. Consequently, any machine state monitoring algorithm that relies on object detection may fail due to image distribution shifting caused by unstable light conditions, unexpected blocking obstacles, or unusual vibration. As mentioned in The second major limitation is that DMM (3D printer interior object detection using YOLO) requires a lot of training samples with manual labeling of ground truths bounding boxes, a common unavoidable problem in the computer vision field. The data collection, labeling and training of the machine learning model may cost some human labor and time when the system gets set up before it can be used effortlessly with trained models.In the future, more use cases (other than 3D printing and assembly lines ) with diThis research proposed the ASAP solution to automate the supervision in manufacturing and significantly reduce the cost of smart manufacturing deployment in SMMs. In contrast, the cost of training AI models has decreased rapidly due to the development of cloud computing technology in recent years. This study developed the AI-assisted Machine Supervision (AIMS) system that provides SMMs with two major subsystems: direct machine monitoring (DMM) and human-machine interaction monitoring (HIM). The experimental results of a case study in 3D printing validated the feasibility of the AIMS system with affordable VPU configuration and evaluated the efficiency of machine supervision for an ASAP solution.The AIMS system will impact smart manufacturing workers by empowering them with actionable intelligence to contribute to decision-making for machine operation management, production scheduling, demand-side facility management for reducing operation cost, and improved productivity for healthy, safe, and accessible manufacturing environments in SMMs. Essentially, the ASAP AIMS system will function as a human-machine system to engage people and systems in complex data management and human-centered workflow automation and control.


Persistent activity has commonly been considered to be a hallmark of working memory (WM). Recent evidence indicates that neuronal discharges in the medial temporal lobe (MTL) are compatible with WM neural patterns observed in cortical areas. However, the characterization of this activity rarely consists of measurements other than firing rates of single neurons. Moreover, a varied repertoire of firing dynamics has been reported in the MTL regions, which motivate the more detailed examination of the relationships between WM processes and discharge patterns undertaken here. Specifically, we investigate‚ at different resolution levels, firing irregularities in electrode recordings from the hippocampus, amygdala, and the entorhinal cortex of epileptic patients during a WM task. We show that some types of (ir)regularities predict response times of the patients depending on the trial periods under consideration. Prominent burst activity at the population level is observed in the amygdala and entorhinal cortex during memory retrieval. In general, regular and bursty neurons contribute to the decoding of the memory load, yet they display important differences across the three anatomical areas. Our results suggest that nonrandom (non-Poisson) patterns are relevant for WM, which calls for the development and use of statistics complementary to mere spike counts. A great deal of work has been dedicated over the last 50 years to the identification and characterization of neural activity observed during working memory (WM) tasks . The abiDespite the multiple reports, a precise characterization of the persistent activity in terms of firing dynamics is missing. In addition, the properties of this activity that allow for an efficient coding of the memory variables are not fully understood. Here, we ask the question if bursty neurons and/or population bursts contribute to the decoding of WM signals. To answer this question, we investigate firing irregularities in the MTL during a modified Sternberg WM task. We utilize a data set of recordings from a region comprising the hippocampus, amygdala, and the entorhinal cortex of epileptic patients . All of the trial recordings have a fixed length of 8 s and they all start with the fixation period (1 s). This implies that when response times are longer than 2 s a part of the neural recordings preceding the response is missing. In many cases, the trials affected are discarded; this is properly noted where required. Trials containing artifacts were also not considered throughout the analysis  the same number of points used for calculating the test statistic (the Pearson correlation or the dissimilarity between the class distributions), we subsampled, within each session, from the combined A population burst is defined as a period of sustained collective activity where the population firing rate exceeds a certain threshold for at least 100 ms. Our procedure closely resembles the one in Below, we refer to some of the analyses presented in the vs 6–8), we ensured that the same number of trials in each condition was utilized for computing the mean and variance of the spike counts. This was achieved by undersampling the condition with the highest number of trials. A different undersampling was performed for each combination of neuron and time step.Mean-matched Fano factors (FF) were calculated to investigate the trial variability of the spiking responses . In contscikit-learn). The mean accuracy score across the 10 test sets was computed. The entire procedure was repeated 50 times, sampling thus different alignment configurations between trials of different sessions. To generate a null distribution of accuracy scores, we shuffled the class labels 500 times within each bootstrap cycle and repeated the procedure described above. One-tailed Decoding analysis was performed on the neuronal pseudo-population obtained by collecting units across all of the sessions and patients. Spike counts were computed per unit in nonoverlapping windows of 250 ms within the 3-s maintenance period, thus yielding 12 points per trial. A curacies . First, The total numbers of units for hippocampus, amygdala, and entorhinal cortex utilized when discriminating the set size were 418, 233, and 308, respectively; when classifying the response correctness, the numbers of units were correspondingly 103, 66, and 58. A large part of the decoding analysis is performed on subsets of these units defined by the To quantify the variation in accuracy when removing an n-tile with respect to the full population, we computed within each bootstrap cycle a d-prime or sensitivity index as follows: All statistical tests were two-tailed unless stated otherwise. The utilization of nonparametric tests  over parametric ones . We adopt a coefficient of local variation while correcting for refractoriness defined as ocument} ; 2009 . CPs are designed to locate and indicate sudden variations in firing rates. We hypothesized that a higher count of CPs will be observed for more irregular, burstier dynamics (see t} test) . NotablyBy construction, en units . In the Having established a substantial independence from firing rate values, we move now on to the specific measurements on Next, we assessed whether the firing behaviour remains stable within a recording session by investigating the trial-to-trial variability of Next, we inspect the relationship between burstiness and the behavioural and trial variables. In preparation, we created a three-way ANOVA model between Irregularities can also occur and be monitored at the ensemble rather than the single-unit level; for this reason, we extended our analysis to include spike patterns resulting from pairs and triplets of units. In practice, for the latter, we combined the spike trains of 2 or 3 simultaneously recorded units and computed the ries see . The ratries see . On the In An interesting behavioural variable yet to be analyzed here is the response time of the patients, which provides a proxy measure for the level of attention as well as for the perceived complexity of the task. In the following, we looked for possible relationships with burstiness values. When simply ordered with respect to the ng trend . We wereFor all of the anatomical areas, higher correlation values are observed almost everywhere for the combinations of 3 units . The difCould the correlations with response times be explained in terms of firing rates? We repeated the analysis using firing rates of single units instead of While the behavior of individual units is informative, it is almost certainly primarily symptomatic of higher order processes taking place at the level of ensembles of neurons compare . Thus, wper se is likely to be evenly distributed across the population. In contrast, the signal associated with the effective memory retrieval (if any) is likely to be captured by a limited set of neurons, such as the one represented by the probe neurons.First, we analyzed the population burst density in a time-resolved manner, and this is shown in vs 6–8 creates results that are generally indistinguishable from each other, posing the question whether the frequency and length of population bursts is more tied to anatomical features than to representation. Thus, we also considered burst composition as a feature. However, in Given the negative results in on event  offers sA suitable measure of the dispersion of our results across trials is not only an important adjudicator of statistical relevance but also an indicator of the presence of heterogeneous biological mechanisms. Here, we turn to a Fano factor (FF) analysis of the variability across trials to answer questions such as, for example, how consistent are changes in population burst activity across trial epochs? We start by examining the average FF values for the 3 anatomical areas along the trial time . The hipWhat is the relationship of FF with the irregularities of the spike sequences? The answer to this question is not a trivial correspondence. For example, for spike trains exhibiting regular patterns, the firing rates could either vary (high FF) or remain constant (low FF) across trials. Similarly, bursty sequences could display either asynchronous patterns (high FF) or coordinated activity during specific epochs (low FF) . If no sTo conclude this analysis, we investigated whether the memory workload modulates the trial-to-trial variability. When discriminating set sizes , we obseIn the final part of our analysis, we asked ourselves whether the decoding capabilities of the population activity observed in  compare . For eacWe show the results of the analysis for the maintenance period in To corroborate these results, we explored a complementary approach to investigate at finer resolution how firing characteristics affect the decoding capabilities. Rather than splitting the whole population into nonoverlapping subpopulations, we remove in turn a small set of neurons equal to 1/20 of the total number characterized. This small set, in the same spirit as before, corresponds to a specific vs 6-8 as for panel (a)). Qualitatively, the results across the quartiles follow the same trends as observed in In the final part of our analysis, we tried to shed light on the previous decoding results by investigating how the single-unit firing rates coded for workloads in the different The discharge patterns that are observed in the MTL during different memory phases are hypothesized to contain information about the memory items as well as shed light on the underlying neuronal dynamics that govern the memory processes. Differently from the approaches adopted in many prior works  see als, we did Our analysis is based in large part on the quantification of the irregularities of spike trains through the ng rates   in the prefrontal cortex , comparaOur analysis shows that the successful retrieval of information in memory (correctness of the response) and the workload (set size) cannot be decoded with clarity from the spectrum of burstiness values of single units or combinations thereof (two or three) during the maintenance period  and S3. In a recent work , it was uartiles .We did observe an important modulation of the population burst activity subsequent to the probe presentation (vs transient).Summarizing, three main results emerge from the analysis of burstiness, which is a measure of the irregularity of the spike trains, during a WM task in the MTL. First, the promptness of the response, but not the workload, is inversely correlated with burstiness levels . The occThe activities of the regions analyzed here  show distinct features in the spiking patterns and in the association to behavioral and trial data. Overall, they indicate that a broader perspective of the timing and structure of spiking patterns, not constrained to the concept of persistent (enhanced) activity, is needed. In our study, we adopted multiple quantities related to spiking variability, such as  neurons . In addi neurons . Also, t neurons . It is l neurons , but alsSupplementary_data_tgac039Click here for additional data file.