Human-Robot Collaboration (HRC) has the potential for a paradigm shift in industrial production by complementing the strengths of industrial robots with human staff. However, exploring these scenarios in physical experimental settings is costly and difficult, e.g., due to safety considerations. We present a virtual reality application that allows the exploration of HRC work arrangements with autonomous robots and their effect on human behavior. Prior experimental studies conducted using this application demonstrated the benefits of augmenting an autonomous robot arm with communication channels on subjective aspects such as perceived stress. Motivated by current safety regulations that hinder HRC to expand its full potential, we explored the effects of the augmented communication on objective measures (collision rate and produced goods) within a virtual sandbox application. Explored through a safe and replicable setup, the goal was to determine whether communication channels that provide guidance and explanation on the robot can help mitigate safety hazards without interfering with the production effectiveness of both parties. This is based on the theoretical foundation that communication channels enable the robot to explain its action, helps the human collaboration partner to comprehend the current state of the shared task better, and react accordingly. Focused on the optimization of production output, reduced collision rate, and increased perception of safety, a between-subjects experimental study with two conditions (augmented communication vs non-augmented) was conducted. The results revealed a statistically significant difference in terms of production quantity output and collisions with the robot, favoring the augmented conditions. Additional statistically significant differences regarding self-reported perceived safety were found. The results of this study provide an entry point for future research regarding the augmentation of industrial robots with communication channels for safety purposes. State of the art automated production cycles today widely use industrial robots. However, most production processes in heavy industries involve human employees at certain points that either coexist or cooperate with these robots. A shared workspace between humans and robots often demands enormous safety precautions, since robots in these contexts usually possess great physical strength combined with high movement velocities . To solvWhile prior HRC-related studies explored subjective measurements, the assessment of objective results are also important. As mentioned before, creating a benefit for production output is necessary for the adoption of HRC in the industry, which is partly addressed in the research for creating an effective task execution scheduling aim for the robot , experimRobots that are deployed in HRC industrial scenarios come in many shapes and forms, ranging from robot arms to more obscure appearances such as the Stewart parallel robot , all desIn addition to the safety restrictions, the realization of an autonomous collaborating robot requires the usage of sophisticated sensor technology that provides the robot with information regarding its environment . Prior rTo address these challenges, we used a virtual reality (VR) sandbox that can be used to create a variety of different HRC scenarios, as the VR technology provides a secure and replicable medium to examine human characteristics when exposed to shared task scenarios involving robots . Prior rIn the following sections, we introduce the theoretical background that provides the basis for the formulated hypotheses and the research question. Afterward, the experimental study including the stimulus material is described, in which different augmentation conditions are compared to explore their impact on production quantity and collision rate. Additionally, based on the results of a prior study , we inveThe current theoretical concept of humans collaborating with robots is derived from the group collaboration between human individuals . EmpiricApart from productivity, another concern for the industry regarding HRC is safety. Currently, potential hazards from the robots are diminished by dividing HRC into three categories: In the first, employees are shielded from the robot either through cages or separated working areas . This enThe anticipated decrease in collisions enabled by the communication channels is also expected to increase the perception of safety within the collaboration task. This can be attributed to the contribution of communication between entities to the perception of safety within a workspace . The saf• H1: Participants produce more pin-back buttons in the augmented condition compared to the non-augmented condition.• H2: Participants collide less with the augmented robot arm compared to the non-augmented condition.• H3: Perception of safety is higher in the augmented condition compared to the non-augmented condition.• Research question: Does the time participants look at the text panel affect the productivity and collision rate?For the purpose of exploring the effect on production capacity, collision avoidance, and the perceived security of guiding and explanatory augmentation of industrial robots in shared task environments, the following hypotheses were formulated:The experimental setup varied the presence vs. absence of augmented communication channels in a between-subjects design where participants were tasked to assemble pin-back button components in collaboration with the autonomously acting robot arm in VR. In the experimental condition, the robot arm was augmented with the three aforementioned communication channels. The non-augmented condition omitted these communication channels.N = 80 , with 40 participants assigned to each of the two conditions. Both conditions contained an equal gender distribution. The average age of the participants was 25 . The majority of the participants were students with a background in computer science and engineering from the University of Applied Sciences Ruhr West.The sample size was To facilitate immersion in the VR sandbox experiments, a virtual environment that emulates an industrial workspace was required. To ensure an authentic depiction, four industry representatives and robot experts were involved in the design process. Qualitative interviews conducted with the experts helped to identify appropriate machinery used in manufacturing plants, the layout of common HRC working arrangements, frequent procedures, and the design of the communication channels. Additional reference material complemented the remarks stated in the interviews , resultiTo ensure stable performance of the virtual environment despite being filled with a variety of props, i.e., pipes, forklifts, and cables. Streaming assets and shader of the objects were optimized for VR usage. This ensured reaching a target rate above ninety frames per second which is crucial for virtual reality, reducing side effects such as motion sickness or eye strain . Non-intThe locomotion mechanic was implemented through the Oculus API and allowed the users to ambulate either through the controller or by their natural body movement. To discourage the exploration of the environment and keep participants focused on the goal of the HRC workplace, the arrangement was enclosed in a separate room that provided a barrier without breaking the internal consistency of the virtual environment. The workplace arrangement itself consisted of a waist-high desk, where the shared task can be executed in collaboration with the autonomous robot arm .The VR sandbox is designed to address numerous categories of collaboration tasks and procedures. For this purpose, a diverse array of interaction mechanics were implemented that allows manipulating actions such as pushing and pulling virtual objects. The usability of the actions was designed according to established third-party applications like the virtual reality toolkit . For theInvestigating several manufacturing processes involving the usage of collaborative robots in real industries, resulted in cumbersome procedures that were deemed too taxing for inexperienced participants. For this purpose, a comprehensible alternative was conceived in the usage of the Badgematic Flexi Type 900 (59 mm) press as a shared task to produce pin-back buttons . The useThe virtual representation of the button press was authentically modeled after the real one, using Autodesk Maya 2018  Autodes. The pinThe shared task itself involved a total of nine individual working steps which were executed alternately between the human participant and the autonomous robot arm  Figure . The proAlthough the VR sandbox was created to allow any form of robot collaboration partner to be evaluated, this scenario used a representation of the KUKA LBR iiwa 7 R800 CR series , which iThe collaboration aspect of the experimental setups within the VR sandbox demanded the robot arm to react adequately towards the actions of the participants. Therefore, the usage of an animation controller that contains a pre-defined set of animated movements was rejected in favor of an inverse kinematic system. This allowed calculation of the required joint angles for the robot arm to reach any target position as well as dynamic movement. Following the structure of the real LBR iiwa series, the virtual model comprised seven degrees-of-freedom (DoF) in a spherical-rotation-spherical kinematic structure using the same parameter as the real robot arm . The inv• Identification: the robot arm is required to detect the movement of the participant represented by the hands and the head of the VR avatar and takes countermeasures to avoid dangerous collisions.• Adaption: The robot arm should adapt to the work pace of the participant and either increase or decrease its movement speed in accordance with the ISO TS 15066 regulations.• Execution: The robot arm can complete its working part of the shared task.• Verification: the robot arm is capable of recognizing that the action of the participant follows the working procedure• Notification: the robot arm is capable to communicate its actions and possible detected deviations from the procedure.To present a wide range of collaboration setups with autonomous robots via the VR sandbox, it was necessary to implement the ability of the robotic arm to perform the collaborative task independently of an external controller such as the Wizard-of-Oz approach. Although the working steps for the robot arm to execute within most collaboration tasks are determined, the actions of the human collaboration partner introduce an unpredictable element, to which the robot arm must react adequately in a functional, predictable or legible way . For theIncorporating the Unity 3D Machine Learning Agents (ML-Agents) in conjunction with the inverse kinematics system enabled the robot arm to conduct these defined characteristics. The ML-Agents open-source plugin provides a framework for the application of various machine learning methods, i.e., reinforcement learning to virtual objects through a Python API and the TensorFlow interface. The ML-Agents SDK itself contains three major components: The first is the agent, which gathers information about the current state of the scene and can execute actions. These actions are determined within the second component, the Brain, which contains the various rules and conditions for the decision-making of each of the corresponding agents. The third component is the Academy, responsible for the global coordination of the simulated environment  Figure .For the ML-Agents framework to assume control over the inverse kinematic system of the robot arm, a Unity GameObject serving as the target for the inverse kinematics-solver was equipped with the provided agent component from the SDK. This agent determined the movement and the speed at which the robot arm heading for its target. The procedure of the shared task with its designated roles was then modeled by adopting the Relational Action Processes (RAP) established by Also, the possibility that the robot arm could be stuck either by the surrounding objects or by a loop had to be considered and counteracted. For this purpose, Unity’s built-in collision system was complemented by a raycast system that sends out radial rays to detect surface meshes of the 3D objects in the vicinity, as the existing Unity collision system only detects entering and exiting collision states. Conducting the learning process of the robot arm without the raycast system would distort the outcome as the ML-Agent framework would not notice states of continuous collision from the robot arm with adjacent objects. A reward system for following the current required target while considering the state of the other agents monitoring the various other items within the virtual environment and punishment for moving away was implemented. Based on the different states of the items necessary for the shared task, these rewards and punishments were adjusted or inverted, enabling the robot arm to follow the procedure for producing a pin-back button in collaboration with human input in this experimental setup . This alThe agents were trained by using recorded data from the collaboration process within the application from nine sessions conducted with three individuals each. A single training segment was defined as the necessary actions for the agents to accomplish the individual working steps of the procedure. The segment was considered to have failed, if the agents reached a collision score of fifty in conjunction with more than eight hundred attempts to reach the respective target, i.e., removing the pin-back component from the press.An expected disadvantage of an agent-controlled kinematic system compared to predefined animations is the potential tremble in the movement due to noise in the training sample. To mitigate this, a per degree movement penalty was implemented to smooth out the motion of each joint of the robot arm as much as possible, ensuring a close depiction of the virtual robot arm’s movement to its real counterpart.An interface component managed the transfer of variables between the agents and the scripts attached to the various non-interactable objects within the environments, such as the display that presented the production quantity to the respective participant. The same approach was used for the three distinct augmentation channels for the communication methods.To evoke the impression of an intended behavior, three distinct unidirectional communication channels were conceptualized. Based on a pre-study , the folThe second augmentation consisted of multi-colored light signals, which were directly attached to the actuators of the robot arm. The concept of these light signals was to alert for potentially dangerous situations with a visual stimulus that is directly in the field of view of the participant and comprehensible at a glance. Derived from suggestions made from qualitative statements from a preceding study , a greenThe third augmentation was the capability of the robot arm to conduct three gestures . Apart from the general approach of providing guidance and explanation, the capability of using gestures was implemented to strengthen the perception of an intended behavior from the robot arm and contribute to the safety attribution of the system. The purpose of the action initiating gesture was to signal the human to proceed with the objective in case no action by the participant was detected. If the collaboration process was stalled through the participant’s inactivity, the robot arm pointed towards the object that was necessary for the subsequent working step . The conTo measure the number of produced pin-back buttons and the collision rate, objective data tracked by the VR application were used. The designated data-set for productivity measured the quantity of pin-back buttons the participant produced in collaboration with the robot arm (H1). The second objective data set detected the number of collisions the participant had with the robot arm (H2). A third objective measure tracked the duration in seconds the participants watched the text panel with the guidance and explanation provided by the augmented robot arm. This measure was only present in the experimental condition, due to the absence of the text panel augmentation in the non-augmented condition.α = 0.69; α = 0.66; α = 0.85; α = 0.72; In addition to the objective measurements, self-reported data were surveyed. The used questionnaire was formulated in German. Items either taken or altered from sources in the English language were translated to German by one researcher and then translated back independently by another researcher to ensure correctness. Measuring the influence of the robot arm’s augmentation on the perceived safety of the participants was done by utilizing self-reported questionnaire data. To measure the perception of safety provided by the augmentation channels of the robot arm (H3), four scales were used. The first contained five items regarding safety aspects of the workplace  in version 22 from IBM was used.F = 12.63, p < 0.01, M = 8.2, SD = 1.40) than in the non-augmented condition  (r(80) = 0.39, p < 0.01).To test H1, an ANCOVA was calculated using the experimental condition as an independent and the production output as a dependent variable and the rating of the augmentation channels, prior experience with industrial robots, and technology affinity (TAM2) as the covariates. Supporting H1 results show a statistically significant difference between conditions ( = 1.53) . The proF = 5.93, p < 0.01, M = 53.57, SD = 47.40) compared to the non-augmented condition  (r(80) = 0.24, p = 0.03) supporting H2.H2 was tested by using an ANCOVA with the experimental condition as the independent and the detected collisions as a dependent variable and the assessment of the augmentation channels, prior experience with industrial robots and technology affinity (TAM2) as the covariates. The results revealed a statistically significant difference separating both conditions (= 81.49) . CollisiF = 5.47, p < 0.01, M = 3.33, SD = 0.59) compared to the control non-augmented condition . The results support H3.The third hypothesis was examined by calculating an ANCOVA that contained the experimental condition as the independent variable and the perceived safety rating of the robot arm as the dependent variable with the safety aspects of the workplace as the covariate. The results indicated a statistically significant difference between the two conditions (SD = 99.61).Results of the Pearson correlation indicated that there was no statistically significant effect between the time participants looked at the display and the collision rate nor the production output with an average display gaze of 253.02 s (With the aim to provide an adaptive and accessible application suitable for HRC experimental studies, we developed a VR sandbox as a modular platform, as described in chapter 3.2. Based on best practices from prior work , every mThe goal of this experimental study was to examine the effect of augmented communication on productivity and safety in shared task setups involving the collaboration between humans and autonomous industrial robots. Prior studies conducted within the VR sandbox focusing on subjective measures revealed various benefits of equipping a robot arm with communication channels in HRC setups . Yet oneIn accordance with the first hypothesis that addresses the number of produced pin-back buttons, participants of the augmented condition generated a higher production quantity compared to the control group. Considering that the assessment of the communicative augmentation strongly correlated with the quantity of produced assets, it can be assumed, that the explanation and guidance provided by the robot arm contributed to participants performing better in terms of productivity. According to human group collaboration research , the comThe second hypothesis stated a reduction in collisions between the robot arm and the participants in the augmented condition. The results support the hypothesis that participants of the augmented condition collided less frequently and that this occurrence correlated with the assessment of the communication channels. Considering that the robot arm’s augmentations enabled it to convey potential hazardous situations through multiple channels, it can be assumed that participants were better suited to recognize these collisions and adapt their behavior to prevent them . AlthougThe third hypothesis complemented the gathered objective measures of the collision rate with the subjective survey to examine if the potential benefit from the augmentations in safety affected the participants’ perception. The results of the experimental study indicate a contribution of the augmentations of the robot arm towards a stronger perception of safety by the participants. With both the perceived safety of the system and the workplace scored better in the augmented condition, a statistically significant difference could be detected. It can be argued that the information provided by the augmentation channels reduced the uncertainty and therefore contributed towards the impression of a safe system . A possiLimitations include the respective constraints of the VR technology, the study design, and the composition of the sample that is discussed in the following.While the study used a sophisticated VR headset, the image resolution of the device still diminishes the visual fidelity of the experience. In conjunction with the limited interaction capabilities of the motion-based controller, the usage of VR can only approximate the realism of a shared task study involving a real robot. Influences that are present in real HRC setups, like touching the robot or the components that are part of the collaboration are omitted in VR, resulting in the absence of a sensory channel that might contribute to the assessment of the situation. However, findings from preceding studies suggest that the participants immersed themselves into the experience and even recognized sudden or unexpected movement by the robot arm as threatening , althougA noteworthy limitation regarding the study design is the usage of the pin-back button press as a shared task. While the usage of a collaborative robot for such a simple task is exaggerated and not appropriate for an industrial context, the relative straightforwardness of the pin-back button machine allowed to establish a comprehensible shared task scenario. Participants independent of prior experience were, therefore, able to execute the procedure and develop a work pace based on the guidance and explanations of the robot arm. Albeit not applicable to complex procedures that are found in industrial manufacturing, the used task allowed to gather insights into the participants’ behavior when exposed to such a scenario. Another limitation in the study design is the short exposure time of the participants with the stimulus material. Considering that industrial employees tasked to collaborate with robots are expected to work with them during prolonged shifts, the dynamic of that relationship that might emerge in this time frame cannot be emulated by the 10 min that were applied in this study. While similar HRC studies are conducted with comparable exposure times for the participants, it is advised to investigate possible deviations from the hereby gathered results in long term studies.Further worthy of mentioning is the composition of the participants. The sample consisted predominantly of students associated with the field of computer science and engineering. Thus, the gathered results do not apply to the general population and in particular to experienced industrial workers. However, because the presented scenario involving AI-enhanced autonomous robots deployed for collaboration can be anticipated for the future, the usage of students that provide the forthcoming workforce can be argued as appropriate.An additional limitation in this experimental study is the moderate reliability of the perceived safety scale used for H3 (described in The concept of complementing the individual skills of human employees with the advantages of robots will become ever so important in industries with increasing competitiveness and dynamic production cycles. However, current implementations of shared workspaces between humans and robots are restricted by necessary safety precautions that limit the areas of application where the combined work of robots and humans can create an economic benefit. Augmenting autonomous robots in shared task environments with communication channels shows promise in enhancing production quantity, reducing collision risk, and perceived safety. These factors play a significant role in establishing HRC in the industry, as only an economical and safe implementation of the concept convinces industry decision-makers to adopt this approach. The results of this study indicate that these augmentations that contribute to actual safety by reducing collisions between the robot and the human collaboration partner, also increase the perceived safety of the system. Nonetheless, the tendency for augmentation for autonomous acting robots to award several advantages to the collaboration process, implicates that HRC-related research and the industry should examine different approaches on how to integrate communication-based augmentation into these work scenarios for upcoming production processes. To cover this subject, the presented virtual reality sandbox application provides the first step for a flexible tool to investigate potential solutions for these essential questions for HRC.


Anticipating pedestrian crossing behavior in urban scenarios is a challenging task for autonomous vehicles. Early this year, a benchmark comprising JAAD and PIE datasets have been released. In the benchmark, several state-of-the-art methods have been ranked. However, most of the ranked temporal models rely on recurrent architectures. In our case, we propose, as far as we are concerned, the first self-attention alternative, based on transformer architecture, which has had enormous success in natural language processing (NLP) and recently in computer vision. Our architecture is composed of various branches which fuse video and kinematic data. The video branch is based on two possible architectures: RubiksNet and TimeSformer. The kinematic branch is based on different configurations of transformer encoder. Several experiments have been performed mainly focusing on pre-processing input data, highlighting problems with two kinematic data sources: pose keypoints and ego-vehicle speed. Our proposed model results are comparable to PCPA, the best performing model in the benchmark reaching an F1 Score of nearly  Road safety is one of the main concerns in the world, being the eighth leading cause of death and the first among young people between 5 and 29 years old. Traffic accidents caused approximately Vulnerable road users (VRUs) represent more than half of all these deaths. Pedestrian Vulnerable Road User (VRU) group is the most affected one in urban roads in European Union, with l(ETSC) .Thanks to various European Union (EU) initiatives and actions, the number of road fatalities has been decreasing since 2011 following a promising and continuous trend. However, it has not been enough to achieve the initial target of Car-to-pedestrian impacts are one of the most frequent accidents occurring in urban environments due to the high level and intensity of interactions between vehicles and pedestrians. Car manufacturers are developing Autonomous Emergency Braking (AEB) systems  to suppoUnderstanding human behavior has been widely studied through trajectory prediction systems  by subtracting the mean and dividing by the standard deviation of the training set.This branch is in charge of encoding a heterogeneous group of features composed by 2D bounding box image coordinates, extracted directly from ground truth annotations, 2D image coordinates of pose keypoints, obtained as the output of a pose estimation algorithm, and ego-vehicle speed, a continuous variable obtained from the ground truth annotations extracted from an OBD sensor. In L layers) are proposed:ViT encoder: similar to the one proposed in [L layer’s output is summarized and the resulting vector is forwarded through a layer normalization and a simple feed-forward layer. Dropout is applied through all the processes, after every feed-forward layer, except for the last one. The diagram of this architecture is shown in posed in  for imagVanilla transformer encoder: using the original proposed encoder in [coder in . The maiThis branch is based on different variants of the transformer encoder architecture proposed initially in . Two difBefore forwarding features into the encoder, a linear embedding is applied to the raw input N, and i corresponds to the position in the embedding dimension, from 0 to d.To account for the order of input data, a positional encoding vector Equation ); vanillEquation ). In thec, which is concatenated to the embedded output of positional encoding. This token is a learnable parameter whose purpose is to represent an aggregate sequence representation for classification tasks. Positional encoding is also applied to it . j traverses the interval The loss function is detailed on Equation . x referWe have used AdamW optimizer  with a fTo accelerate the experimentation, we included an early stopping callback, focused on validation loss, for an early over-fitting detection during training. We perform a validation epoch every All the experiments using TimeSformer and most of those using RubiksNet have been performed using a single NVIDIA A100 GPU with 40 GB of memory and an AMD EPYC 7742 64-Core CPU @ 2.25 GHz.We have chosen to use PyTorch Lightning  for all Before validating our proposed model in the previously discussed benchmark, we performed an ablation study concerning data pre-processing techniques. The objective of this study is the analysis of the importance of each input feature and also the importance of the pre-processing strategy. After this analysis, we have performed a set of experiments to see the importance of combining data for training and the generalization ability of our models. Concerning the model architecture, we performed an ablation study for video backbones and transformer encoder architectures. Finally, we evaluated the performance of our models in the benchmark.We performed several experiments concerning the input data to measure the importance of each feature and the pre-processing method applied to each of them. In addition, to measure generalization capabilities, we performed a study of the improvement achieved by our model if trained on the combination of both datasets in the benchmark.local box, where the bounding box is cropped and padded (black regions) to maintain the aspect ratio; local context, where the longest edge on the image is enlarged by a factor and the shortest is also enlarged to reach a square shape; local surround is the same as the previous one but deleting the image information inside the bounding box of the pedestrian and replacing it with gray pixels. In the benchmark, bounding boxes coordinates are used to crop the pedestrian region from the full image frame, following different strategies: local box warp  with thCenter coordinates and height: we obtain the center coordinates of the bounding box and its height. We did not include the width to avoid redundancy, as we include it as a measure related to the distance between ego-vehicle and pedestrian;Center coordinates and height including speed: in addition to the above features, we include the speed of change of center coordinates and height.Additionally, we performed experiments with two variants of these coordinates :Pose keypoints provided in the benchmark are obtained offline from a convolutional pose estimation method. Since it is an error-prone method, we want to evaluate whether this input feature is helpful in any way.We did not include ego-vehicle speed in our data experiments, only in the benchmark evaluation for a fair comparison with state-of-the-art models. We hypothesize that this feature is highly correlated with pedestrian action, as pedestrians tend to cross when the ego-vehicle stops and wait when the ego-vehicle does not stop. Because of that, a reduction in the ego-vehicle speed triggers the crossing action in most of the positive cases and it happens before the pedestrian decides to cross. For this reason, ego-vehicle speed should be treated as an output of an AV and not as an input to one of its perception modules.Due to the ego-vehicle speed controversy and the automated nature of pose keypoints, we performed a grid analysis of all the possible combinations of input variables.Horizontal flip: apply a random horizontal flip on the image plane;Roll rotation: apply a roll rotation of the 3D sequence, which is equivalent to applying a 2D rotation on each image in the sequence;Color jittering: apply a random change in brightness, contrast, saturation, and hue of the input sequence.Under the hypothesis of the scarcity of training data in comparison with other general action datasets, we experimented using three types of data augmentation for video sequences. If the augmentation is applied, the same change is performed in all images of a sequence:Deep learning models perform better with more varied training data, becoming models prone to task generalization rather than over-fitting. To test this property, we performed two different training schedules: one model only trained on the PIE dataset, and another one trained on both PIE and JAAD. We tested both models on both datasets test sets.We performed two groups of experiments, focused on the video backbone and the transformer encoder used in the kinematics branch. For the first group, we explored different variants of RubiksNet and TimeSformer, pre-trained on different datasets. In the case of the transformer encoder, we experimented with different encoder types and different summarizing strategies.For RubiksNet backbone, we have used a variety of pre-trained weights, ranging from RubiksNet-Tiny variant (2 (SSv2)  and Kine2 (SSv2)  datasetsowTo100M  (HT100M)owTo100M  (K600).After several experiments with all of the above-mentioned backbones, we found that SSv2 pre-trained models outperformed the rest by a large margin. One of the possible reasons is the training schedule followed with the rest of the datasets. We also trained the backbone from scratch, but SSv2 pre-trained model also outperforms this strategy. For this reason, we have chosen this backbone in all our experiments.Two different transformer encoder architectures are proposed in this work. In addition, its output is processed using the mean operation over the temporal dimension or by applying a flattening operation, resulting in a one-dimensional vector. By combining the encoder type and the output summarizing strategy, we obtained four different options. The best performing option is also compared with the model without a kinematic branch, to see if this branch improves the result.After performing all of the previous experiments, we have found which pre-processing strategies improve our model’s performance and also the importance of the different input features.Multi-stream RNN (MultiRNN) [ultiRNN) : Two strC3D [C3D : 3D convInflated 3D (I3D) [3D (I3D) : 3D convPCPA [PCPA : best peWe have compared our model variations with four different models ranked as the best-performing ones in the benchmark:However, for a fair comparison with PCPA, the best model in the benchmark , we havelocal box cropping strategy and without any normalization, respectively. Our combined features transformer encoder have the following hyperparameters:Query, key and value size is the same Number of self-attention heads Number of transformer encoders Multi-layer perceptron hidden layer dimension Dropout rate, applied after embedding and the MLP block To check if this performance similitude is due to the transformer encoder and not to the video backbone, we developed our best transformer encoder in Tensorflow and changed the recurrent encoders in the PCPA model with it. All non-image (kinematics) inputs are concatenated and forwarded through it after embedding them. The rest of the PCPA model remains the same, including the video 3D backbone and the fusion part. However, we changed the optimizer to AdamW and tuned the hyperparameters of our model to accelerate training. As AdamW applies weight decay, we deactivated direct regularization on layers and applied a weight decay of PIE dataset used for training;Batch size of 16 samples;Local box warp used as the pre-processing for bounding boxes crops;TimeSformer used as backbone, pre-trained on SSv2 and fine-tuned. The output vector size of 1024;Fusion strategy: concatenation;Input sequence length Learning rate with value Input image dimension of Weight decay of Unless specified, we performed all of the previous detailed experiments with the same hyperparameters, to see the effects of data modification on the results. These experiments are not focused on reaching the best performing metrics but on reaching average results in a short period. This default set of hyperparameters is the following:Due to the difficulty of the task given and the stochastic nature of models and training algorithms, we gather results of every experiment using a set of eight different random seeds, displaying on result tables the mean and its standard error, to show the discrepancy between the group of eight samples and the real distribution mean.For the benchmark models, we fixed random seed to 42 in both PyTorch and Tensorflow implementation. We have included as We have used the same metrics available in the benchmark. Accuracy does not represent a good performance estimator in imbalanced problems, so we focused our analysis of the results on F1 score (F1) and the area under the Receiver Operating Characteristic (ROC) curve (AUC). F1 Score is the harmonic mean of precision and recall. It is used as a measure in which both metrics have the same weight. However, in real applications, higher recall is more important to improve safety and higher precision is better to avoid false positives and allow smoother driving, decreasing unnecessary braking. For this reason, we have also included precision and recall. Finally, AUC represents the ability of our classifier to distinguish between both classes. A value of 0.5 means that the classifier behavior is equivalent to randomly choosing the class. Metrics are calculated for the positive class (crossing case) using the scikit-learn library .The results for each cropping strategy in local box cropping strategy, are better for the results than applying data augmentation . As a lighter alternative, we performed experiments with the RubiksNet tiny version, which in combination with the rest of the model, contains only To see if the transformer encoder for kinematics features was indeed the reason for these good results, we included this encoder in the PCPA model, substituting recurrent ones. We trained these models with only bounding boxes image crops and coordinates, obtaining comparable results to the ones obtained with TimeSformer and RubiksNet, and also with some of the PCPA variants. For a fair comparison, we trained two PCPA variants with the same random seed used in our model. The first case uses the best-performing model in the benchmark and the same training procedure. We can see that there is a big difference between the final results . These new features will be 2D or 3D, depending on the availability of datasets in the literature;Experimentation with different data cleaning strategies in training time, applying a maximum occlusion level, pedestrian minimum size, etc.;Simulate real case scenario to find new weaknesses and strengths in available models.In the future, some possible lines of work are detailed below:


Background: As the effectiveness on stress urinary incontinence (SUI) prevention of pelvic floor muscle training (PFMT) for pregnant women has been inconclusive, we are planning to conduct a trial to evaluate a video program designed for prevention of SUI developed through combining PFMT with global postural reeducation (GPR). Methods: As a randomized controlled trial, eligible participants will be randomized (1:1) into an exercise group and a control group to perform PFMT regularly following video guidance or with no intervention, respectively. The experimental stage will be from the 16th gestation week (GW) to the 12th month postpartum, with eight appointments at the 16th, 28th, 37th GW, delivery, the 6th week and the 3rd, 6th, and 12th month postpartum. Data will be collected regarding urinary leakage symptoms, the stress test, the modified Oxford Scale, pelvic floor ultrasound, perineal laceration classification at delivery, neonatal Apgar score, and questionnaires . The primary outcome is the occurrence of the symptomatic SUI and positive stress test at the 6th week postpartum. Discussion: This protocol is anticipated to evaluate the efficacy of the intervention via video app for the design of a future randomized control trial (RCT). Trial registration: The trial has been registered at Chinese Clinical Trial Registry (registration number: ChiCTR2000029618). Stress urinary incontinence (SUI) is defined as “involuntary leakage of urine when abdominal pressure increases” . The prePelvic floor muscle training (PFMT) has been recommended as the first optional treatment of SUI , while tGlobal postural reeducation (GPR) was projected to correct postural misalignments by stretching the muscle chains. As the spine is a continuous boney structure that is closely associated with abdominal and pelvic dynamics, it is commonly recognized that spine restructure through posture reeducation would benefit the functions of the PFMs. Our previous study demonstrated that the spinopelvic skeletal shape had a close relationship with pelvic floor dysfunction . In 1981To explore a more effective exercise regimen for SUI prevention, we developed a training program through combining PFMT with GPR training (PGT program) under the collaborative efforts from several urogynecologists and sports experts from Peking University.Patient’s compliance is another important factor for the effectiveness of PFMT. Face-to-face training is time-consuming and needs frequent visits to relevant facilities, which therefore further decreases the adherence to the training program. In addition, it is quite hard to organize face-to-face training programs during the COVID-19 pandemic. For a long time, what has been bothering the urogynecologists is how to achieve intensive supervision and encourage participants to keep high compliance. Popularization of mobile phones makes almost everyone accessible via mobile devices, which enabled women to connect to the world from at home; thus, a PGT video guidance on mobile apps can create potentiality to enhance the adherence and compliance of the patients who can perform PFMT and GPR at home.Therefore, an app-based video program was developed with the collaboration of software engineers. The video program tends to guide the patients to do regular PFMT and GPR at home or any private room. To make the patients’ compliance evaluable during the COVID-19 pandemic, we integrated a high-intensive online supervision module into the video program. We conduct this multicenter RCT to evaluate the effectiveness and safety of the video program on the prevention of postpartum SUI. The study is designed as a two-arm, parallel, RCT designed following the Consolidated Standards of Reporting Trials (CONSORT). Participants will be recruited from the obstetrics departments of ten hospitals (the centers) including Peking University People’s Hospital, Peking University International Hospital, Peking University Shenzhen Hospital, Beijing Obstetrics and Gynecology Hospital, Fangshan Maternal and Child Health Hospital of Beijing, Fengtai Maternal and Child Health Hospital of Beijing, Mentougou District Hospital of Beijing, Zhengzhou Central Hospital Affiliated to Zhengzhou University, Luohe Central Hospital of Henan Province and The First Obstetrics Hospital of Shanghai. The ten hospitals are selected as the centers because (1) they have enough patients to recruit the minimum number of patients proposed for each center, and (2) the investigators at those centers are knowledgeable and experienced in the identification and treatment of the SUI. Pregnant women will be eligible for recruitment if they are: (1) 20 to 40 years of age, (2) primipara with a singleton pregnancy of less than 16th gestation week (GW), and (3) willing to participate in the study and understand the research procedure well. In the first enrollment examination, participants will be excluded if they were confirmed: (1) having serious complications  that may be aggravated by the intervention, (2) having SUI or pelvic floor organ prolapse (POP) according to pelvic organ prolapse quantitation (POP-Q) staging, (3) having history of cervical insufficiency, recurrent miscarriage, and induced labor, (4) unable to finish all follow-ups, (5) infecting COVID-19, and/or (6) unable to provide consent in writing . Each paA randomization list will be created by a statistician from Peking University for each center before the start of the research using SAS 9.4 . The numbers of participants in EG and CG are balanced, with a 1:1 ratio, and put into blocks of four. A unique study ID will be given to each participant. The randomization result for EG and CG are sealed in the ID-matching envelopes. The grouping details will be concealed to the investigators who will account for group allocation, recruitment, informed consent, and data collection. Eligible women are informed of grouping allocation via opening the sealed envelope after signing the informed consent. It is not feasible to blind to participants, urogynecologists and physiotherapists engaging in the intervention. However, all the investigators to perform patient evaluation and data analysis will be masked.All participants in EG and CG will have same routine obstetric prenatal care following the standard clinical procedures. No intervention will be given to the participants in CG. Participants in EG will perform PGT from the 28th GW to their delivery following the video guidance on mobile app. They will be instructed to download and install the app by research assistants.The app provides health education and a PGT video. Watching the health education video, participants can learn the pelvic floor anatomy and function and are informed about the negative impact of pregnancy and delivery on PFM. The PGT video includes PFMT and GPR programs designed collaboratively by the urogynecologists and the sports experts. The GPR combines global posture and respiratory training with PFMT to enable participants to exercise PFMT while posturing.PFMT mainly focuses on voluntary contraction and relaxation of PFM around urethra, vagina and rectum. It provides instruction on coordinating PFM contraction and relaxation by exhalation and inhalation of a breathing circle, respectively, while avoiding the involvement of abdominal muscles. Participants need to contract the PFM first with moderate strength and upholding for 6–8 s, followed by 6–8 s PFM relaxation (the primary PFMT procedure), then repeat primary procedure five times to train the type I muscle fibers. As the second step, participants are asked to do quick maximum PFM contraction and relaxation in 1 s for five times, followed by 10 s of relaxation (the secondary PFMT procedure). The secondary PFMT procedure will be repeated for eight times to train the type II muscle fibers. Both the primary and secondary PFMT procedures need to be repeated for three sets in an exercise circle. Participants in EG need to perform PFMT twice a day following the instruction of the video program. The GPR requires the participants to perform respiratory training, PFMT, and global posture simultaneously. It consists of four sets of exercises, each lasting about 30 min and composed of four groups of postures. The postural intensity increases from the first set to the last one. The physiotherapists will instruct the participant to do the proper set of exercises at a specific period according to her gestation weeks. Respiration training during the whole exercise is essential to expand the chest and abdominal cavity that aims to strengthen the relevant muscles. Participants will be guided to exhale deeply while contracting muscles and inhale deeply while relaxing muscles. The program starts with respiratory training in the supine position while performing one set of PFMT, followed by 20 min of aerobic exercise by keeping global postures such as bridge pose, cat pose, child’s pose, and mountain pose, and in-between one set of PFMT is performed. The exercise of GPR will be ended with meditation in the supine position and one set of PFMT will be performed simultaneously. The twice-a-week GPR was originally programmed to be performed once following guidance of the app at home and once under face-to-face guidance of the physiotherapists. In consideration that it is not safe to organize face-to-face guidance in training rooms during COVID-19 pandemic and the need to regulate the postures and procedures and collect data to evaluate the compliance of the participants, we modified the protocol to organize video-trainings for EG participants to perform GPR under the living guidance of the physiotherapists on-video via Zoom meetings once a week. Four Zoom meetings will be scheduled for video training in each week, each focusing on one of the four exercise sets. An individual participant will be invited to visit one of the Zoom meeting to practice the exercises appropriate to her GW. Each video-meeting can be divided into several classes according to the number of participants, with each limited to 10 persons, so as to facilitate the physiotherapists to give enough care to each participant and to maximize the effectiveness. The site assistants will notify each participant the Zoom link specially for her in each week in advance, but keep her blind of the other Zoom meeting details to avoid protocol bias. The site assistant will also record the attendance of each participant in terms of the attendance time, the procedural and intensity compliance, and the adverse events, etc. Any participant absent at the scheduled class will be reminded to join the next class on the same exercise set so that she can “catch up” the scheduled one in the following Zoom meeting. The scheduled class will not be cancelled even with only one participant. The role of the physiotherapist is to guide participants in the class to accomplish the full exercise set and to regulate any inappropriate acts. To protect participants’ privacy, no video and audio records are taken and preserved during video-conference, which are fully informed to and consented by the participants.Any pregnant participant experiencing abnormal symptoms  will contact the site assistant in each hospital, and the assistant will refer them to the obstetric department as soon as possible, followed by an assessment of whether they should continue the intervention or not. An anticipated adverse event report will be prepared and sent to the principal investigator (PI). Data regarding the clinical examination during the follow-ups will be collected from all participants in both groups. A questionnaire of demographic characteristics including the age, body mass index (BMI), educational status, working type, working posture, toilet type, PFMT history, constipation history, smoking history, number of gestations, number of abortions, family history of SUI, and family history of POP will be collected from the participants with gestation ≤ 16 GWs. The Prolapse/Urinary Incontinence Sexual Questionnaire short form (PISQ-12)  will be Four follow-ups will be conducted on all participants respectively on the 6th week and in the 3rd, 6th and 12th month postpartum. To conduct the multiple physical valuations in a time-saving way, it is proposed to check the participant in the following sequence: first, check the stress test, POP-Q, MOS when the bladder is half-full; second, conduct the pelvic floor ultrasound examination and the pelvic floor electrophysiological test  with the emptied bladder; and last, fill the PISQ-12. The pelvic floor electrophysiological indicators mainly include the vaginal resting pressure (cmH2O), the vaginal maximum contraction pressure (cmH2O) and PFM fatigue (%). I-QOL, ICIQ-UI SF, and OABSS will be provided to participants who show symptoms of urinary leakage or UUI as appropriate .Gynecologic examination, pelvic floor ultrasound, pelvic electromyography, and fetal safety will be evaluated respectively by the urogynecologists, ultrasound professors, electromyography nurses, and obstetric professors in each hospital, who will be trained in the Peking University People’s Hospital before the launch of the study. A program that is preset into the video will record times for the full-playing of the video and automatically transfer the data into the database as the training frequency for each participant. The site assistants will confirm the training frequency of each participant in EG every week by checking the records from the online-guided software, exercise diary, and the video-training. Accomplishment of PFMT will be identified for each participant should she accomplish more than 80% of the training programs prescribed for every week. Data input devices will be provided to the hospitals to be involved in the research. Paper files will be prepared and kept for each participant to record all data to be collected in the full study stage. Exercise diaries from participants in EG will be used to evaluate their PFMT frequency and training intensity. The database will be protected from access by a password, and the final dataset will only be accessible to the PI, and, under authorization of the PI, to the study coordinator. Desensitized data are available for research purpose only per the permission of the PI. The primary outcome of this study is the occurrence of SUI at the 6th week postpartum, which will be diagnosed based on urinary leakage when abdominal pressure increases or positive stress test. A positive result of the stress test is defined as involuntarily urinary leakage during Valsalva or coughing in the lithotomy position.The secondary outcomes include: (1) the occurrence of SUI at the 12th month postpartum; (2) the PFM strength at the 6th week and 12th month postpartum, which will be measured with the MOS and pelvic floor electrophysiological test.The sample size was calculated using PASS 2019 software. Qi et al. [Statistical analysis will be conducted using SPSS version 23.0 and the assumed significance level was 5%. Categorical variables will be described as numbers and percentages. If continuous variables followed a normal distribution, data will be represented as means ± standard deviations (SDs). Otherwise, medians  will be calculated when continuous variables do not follow a normal distribution. For the primary outcome, we will compare the occurrence of SUI at 6-week postpartum between EG and CG by chi-square test or Fisher’s exact test as appropriate. For secondary outcomes, the counting data will be analyzed by logistic regression and the quantitative data will be compared by linear regression as appropriate.Statistical analyses will be conducted on an intention-to-treat (ITT) basis by the statistician at Peking University Clinical Research Institute. Data analyses of EG will also include those regarding the participants’ adherence to the protocol with at least 80% participation to be the indicator of task achieved.The outcome analysis will be conducted with multiple imputations to take missing values into account. The statisticians will perform sensitivity analyses of the outcome first by imputing missing values as failures and then by using complete cases.Pregnancy and delivery are important risk factors in the development of SUI because it potentially weakens and damages nerves, PFM, and connective tissues ,19. The The correctness and compliance of PFMT may be the key factors for the result inconsistencies of those studies ,7,8,9,10Twenty-seven years after Souchard  first prPetros  has founAs we all know, mobile phones are becoming increasingly common, thus the app-based video guidance is easy to achieve with the aid of mobile devices. During the COVID-19 pandemic, adoption of the mobile-device-based PGT video program and Zoom meetings will not only enable the participants to perform PFMT and GPR safely at home, and increase participants’ interests for exercise, but also improve the compliance of participants, because participants can easily perform the 10-min PFMT session in any spare time and at any place. The automatic frequency recording design of the PFMT software is expected to provide better supervision on the compliance. This protocol is designed according to referring guidelines, consensus and high-quality studies. As a multicenter RCT, it will involve ten hospitals including tertiary hospitals and maternity hospitals located in several provinces, purposed to achieve more convincing results than other single center protocols ,32. The possible limits of this trial are that there will be no comparison of the virtual guidance and supervision versus the face-to-face ones because of the COVID-19 pandemic, and that the data may be contaminated due to no effective way to limit the participants in the control group doing PFMT personally. Another limitation of the trial is that the long follow-up time may increase the amount of the missing data.In conclusion, this study is expected to help us understand the efficacy and feasibility of the PGT video app and the intervention design for future clinical trials.


Our study suggests that more focus should be directed towards the exploration of behavioral models for hominins, as directionality may be a more powerful factor for crossing a barrier than environmental opportunities alone. A prerequisite for this is to perceive the opposite shore. Furthermore, to provide a comprehensive understanding of hominin expansions, the CSR allows for the integration of results obtained from small-scale simulations into large-scale models for hominin expansion.Understanding hominin expansions requires the comprehension of movement processes at different scales. In many models of hominin expansion these processes are viewed as being determined by large-scale effects, such as changes in climate and vegetation spanning continents and thousands or even millions of years. However, these large-scale patterns of expansions also need to be considered as possibly resulting from the accumulation of small-scale decisions of individual hominins. Moving on a continental scale may for instance involve crossing a water barrier. We present a generalized agent-based model for simulating the crossing of a water barrier where the agents represent the hominin individuals. The model can be configured to represent a variety of movement modes across water. Here, we compare four different behavioral scenarios in conjunction with a set of water barrier configurations, in which agents move in water by either paddling, drifting, swimming or rafting. We introduce the  Homo first appeared in Africa 2.8 Ma [Homo species differing in cognitive and technological capabilities. Many of those expansion phases required the hominins to cross water bodies of varying sizes, from large rivers to sea straits [The genus a 2.8 Ma  and even straits . The ext straits –10 and aThe purpose of this study is to introduce a generalized agent-based model that simulates hominin movement across a hypothetical, simulated landscape divided by a water barrier. We distinguish the ability of hominins to cross water barriers in the form of scenarios. In the model we therefore speak of agents instead of a particular hominin species. These represent levels of physiological and cultural performance, expressed as water movement skills. The model can be configured to represent a variety of water movement skills. Here, we compare four scenarios ranging from paddling to rafting.Our study presents the first agent-based model for simulating small-scale hominin sea crossing. We explore the technical functionality of the model in a pilot study, which can be transferred to geographically explicit settings in subsequent steps, and test the features of the model as a prerequisite to distinguish the effects evoked by the model design from the results of behavioral alternatives for the agents. The results will then be integrated into large-scale simulations in the future. With the agent-based simulation, we provide an experimental context to the numerous conceptual models for hominin sea crossing from archaeology and paleoanthropology e.g., , 11–14).–14.11–14Hominin expansion is generally represented as a large-scale spatio-temporal process being driven by demographic and/or environmental factors and in general, as following a diffusion-like mechanism –24.Models involving low spatial resolution favor terrestrial routes over those involving the crossing of water barriers, sometimes to such an extent that non-terrestrial routes are entirely omitted from further consideration. In addition to the spatial resolution, the temporal resolution determines which dynamic processes are represented in the model. For example, movements across water barriers are reduced to a “yes” or “no” decision which must be decided prior to the modeling procedure –20, 23. In the model, we simulate the movement of agents on land and across a water barrier and quantify this process by utilizing the output variable, CSR. When describing the model parameters, we use the terminology following Montgomery and Douglas  where thThe map of the environment in the agent-based model consists of a grid of “patches”. A water barrier deep and wide enough to prevent foot wading separates the source and target areas. Patches are defined by three features: resource quality, slope and water .2 of geographical space. Factors for creating the map are given in Resource qualities refer to the availability of resources for subsistence on land and are represented by a float value of between 0 and 1. A value of 0 means that the patch possesses no resources and a value of 1 means that the patch has resources with maximum quality. Resource qualities of between 0 and 1 represent intermediate amounts. The slope is represented in percentage rise. One patch represents 1 kmHomo erectus, were of sufficient body size and proportions [Hominins, such as portions –29 to haportions –32, evenThe agents represent hominin individuals who perform movement on land and in water , wherebyFurthermore, they can leave the map or die. Initially, 1000 individual hominin agents are placed randomly within the source area corresponding to population density estimates for hominins .An agent who is placed on the map selects a target patch to which they direct their movement. The agent perceives neighboring patches within a radius of 20 km. Within this radius, the agent selects a target patch towards which they will head. A higher radius of perception increases the sample of potential target patches surrounding the agent. The radius of perception allows the inclusion of environmental factors such as weather, wave action, clouds, fog and darkness, that affect the perception of the environment into a single value. By default, the agent will select the patch with the highest resource quality with a probability of 90% or a random patch within the perception radius with the remaining 10% probability. Our model, therefore, allows for some degree of decision uncertainty. Due to the random distribution of resource qualities and slope values, the movement on land in the generalized model results in a randomly directed movement with randomized walking speed. We note that the movement would not be random on a map with heterogenous distribution of resource qualities and realistic topography, such as it would be the case in a geographic explicit map. In the generalized model introduced here, the movement on land acts as place holder for more sophisticated movement models (see section “Crossing decision and how to get into the water”).In order to reach the target patch, the agent moves with a certain movement speed that is determined by the optimal walking speed and the slope value of the local patch. We assume an optimal walking speed of 5 km/h for the agents which closely correspond to the average optimal walking speed of 1.5 m/s of present humans . The effOn water patches, the agents move by paddling, drifting, swimming or rafting. However, the particular movement mode on the water patches is defined by the specific behavioral scenario described in the section below. In each of the scenarios, the movement on water patches is affected by water current. The water current modulates the performance of paddling, drifting, swimming and rafting by deflecting the direction in which the agents move. The effect is modeled by adding the current speed, with a west to east direction, to the vector of the hominin agent’s movement.All agents are equipped with an energy storage of 45,000 energy-units. The rate of change in the energetic levels of the agents may differ from empirical caloric  values during a simulation run. Therefore, we chose the term “energy-unit” to describe the energetic level of the agents. As we focus on the movement in water, we apply a simplified definition for the movement on land. Agents do not spend energy on land; we assume that agents on terrestrial patches receive a sufficient amount of resources and so do not die. This, however, is not the case when on the water patches.Hh is the energy loss from thermoregulation in energy-units per hour and Tw is the water temperature in degrees Celsius.On a water patch, an agent dies of exhaustion when their energy level drops to 0. An agent also loses a basic amount of 62.5 energy-units per hour, corresponding to 1,500 energy-units per day. The equivalent caloric demand is assumed for hominins . A dailyIn addition, paddling or swimming reduces the energy storage by a rate of 800 energy-units per hour, corresponding to the caloric demand of English Channel swimmers , whereasAn agent can only survive water temperatures significantly below body temperature for a limited amount of time . An agenAs the water barrier consists of salt water, agents may die of dehydration. In the literature, the estimated survival time without a supply of freshwater is, on average, 100 hours, ~ 4 days . Thus, wAgents who survive the movement on water patches may arrive at a land patch. Upon arrival, their energy storage is replenished to its maximum value of 45,000 energy-units, irrespective of the resource quality on the patch, while the level of dehydration is reset to 0. Aside from dying, the agents may also leave the map. This occurs when an agent moves on a patch at the border of the map, either by a movement decision or by deflection by the water current. Agents who die or leave the map are subsequently replaced in corresponding numbers; new agents appear on a random patch in the source area. Consequently, the total number of agents remains constant .CSR is the crossing success rate after n hours, ft is the number of times a founder group is established and at is the cumulative total number of attempts of water crossing. The number of attempts to cross the water barrier correspond to the total number of agents who move from a land patch of the source area to a water patch divided by the founder group size. A simulation time step (“tick”) represents one hour. We ran each simulation experiment for 90 days .As a tool of analysis, we monitor the CSR to quantify how successful agents have been in crossing a water barrier. A certain number of agents (a threshold) reaching the opposite shore is required before the crossing counts as being successful. A founder group consists of a minimum number of agents (n = 50) who are simultaneously present at the opposite shore. The success rate in crossing a water barrier (CSR) is determined as:Here, we refer to the concept of a founder population following Mayr –44. In tCSR represents a proportional value and varies between 0 and 1. If the CSR is 0, no founder group is established during the simulation run. Although some agents may have accomplished the crossing, the required minimum number of agents for a founder group is not reached. Whereas, if CSR is 1, all attempts of crossing the water barrier result in successful founder groups.We set up experiments in order to test the effect of selected model factors together with four different modes of movement in a systematic way.The model consists of 18 factors that affect the features and behavior of the agents or the environment . We selen simulation runs, dn, is based on the estimation of the confidence interval and is defined as:n is the number of simulation runs, tn−1,α/2 is the student t-distribution quantile, Sn is the cumulative standard deviation, and water-barrier-size), speed of the current (current-speed) and temperature of the water . In the second category, we summarize the factors of perception; these determine to what extent the agents can assess the resource qualities (decision-randomness) and perceive their environment (perception-radius). In the third category, we summarize the factors that determine the physiological conditions of the agents in terms of dehydration (dehydration and hours-until-dehydration), exhaustion  and hypothermia (hypothermia) when moving in the water. In the fourth category, we summarize the demographic properties of the founder group (number-founder-agents), the number of agents on the map (population-density), and the random death rate in the water (death-rate-in-water).As the model contains sources of stochasticity, i.e. a randomized distribution of resource quality, a random initial placement of the agents in the source area and a certain degree of randomness due to the factor ‘randomness of decision’, it is necessary to determine the number of simulation runs required for capturing the variability of the CSR response to ensure statistical validity of the results. For this purpose, we perform a replication assessment with the default levels , as descwater-barrier-size) to test the distances being crossed in a particular behavioral scenario, by varying this factor, we may estimate to what extent a water barrier of a certain size impedes successful crossings. Secondly, we vary the radius of perception (perception-radius) to test different distances for perceiving the resource qualities, by varying this factor, we may estimate the effect of reduced or increased distances of perception. Thirdly, we vary the hours until dehydration (hours-until-dehydration) to test different durations of survival without drinking water while trying to cross the water barrier, by varying this factor, we may estimate the effect of a reduced or increased time of survival without drinking water on the crossing success. Fourthly, we vary the number of founder agents (number-of-founder-agents) to test various thresholds for the number of agents required to be simultaneously present in the target area for the crossing to be rated as being successful. Thus, we are able to quantify the decrease of crossing success along with increasing numbers of founder agents, and the increase in crossing success with decreasing numbers of founder agents.The factor levels of the sensitivity experiments are listed in perception-radius) and extent of the water barrier (water-barrier-size) simultaneously to analyze the combined effects between different ranges of the perception of resource qualities and different distances between the source- and target shores .movement-in-water-direction), the speed (water-movement-speed), energy loss (energy-loss-swimming), hypothermia (hypothermia) and thermoregulation (thermoregulation). We test four behavioral scenarios that represent different water movement skills concerning water crossing performance and navigation, ranging from randomly directed paddling and drifting to controlled swimming and rafting.The model allows representing a variety of behavioral scenarios by configuring the direction of movement in the water  with the default levels of the factors  and 200 water-barrier-size), radius of perception (perception-radius), hours until dehydration (hours-until-dehydration) and the number of founder agents (number-founder-agents). With our results we are able to demonstrate how these factors affect the CSR (The one-factor sensitivity experiments that we conducted consisted of the extent of the water barrier ( the CSR .For the default scenario (Scenario A), increasing the extent of the water barrier decreased the CSR , exhaustion  and hypothermia (hypothermia). In Scenarios A-C the agents are not affected by the hours until dehydration because they die of hypothermia before dehydration even comes into play  and the number of individuals required for a founder group (number-founder-agents). Our model allows to test the performance of different founder group sizes. Our results suggest that a larger founder group is less likely to be established than a smaller founder group. We note that for a permanent occupation, such a group must not only be able to survive the water crossing, but also to exploit the resources in the new area and reproduce. Therefore, in addition to the mere number of individuals in the group, the composition of the group with regards to the ability to thrive in the new environment needs to be considered. This limits a successful founder group to consist of a sufficient number of reproducible adults. The group also has to adapt to the new environment of the opposite shore, i.e., by adjusting foraging strategies to different types of resources and properties of the landscape.Demographic requirements determine the likelihood that a founder population may establish on the opposite shore. In our model, the demographic factors are represented by the density of the population  and to integrate these into critical regions of the large-scale model to provide a comprehensive explanation of hominin expansions across different scales.It is worth noting, that the CSR does not quantify for how long a founder group establishing at the opposite shore is able to survive in the new environment and whether the success in crossing persists across large temporal scales of several thousands of years. Nonetheless, a founder group represents a prerequisite for sustainable settlements at the opposite shore.Along the same lines, the CSR itself does not characterize particular routes across the water barrier. While this is trivial in a uniform water barrier, as presented in our study, it may be relevant in environments where island hopping is possible and, thus, the agents would have choice of different routes.The movement on land is deliberately kept at a basic level because the model focuses on the movement in the water. Therefore, the movement on land, as it is implemented in the model, functions as a place holder for more sophisticated mobility and subsistence strategies. In this basic model of movement, the direction is decided upon the resource qualities within the radius of perception and the speed is determined by the slope value. Nonetheless, the land movement in the source area is not monitored by the CSR because the CSR calculation commences with the crossing attempts when the agents move from a land patch, in the source area, to a water patch. Thus, the CSR is only intended to evaluate the crossing of the water barrier and its success in the target area. As already discussed, the agents need to perceive the opposite shore in order to attempt a crossing. Avoiding low water temperatures is crucial because even a difference of a few degrees Celsius severely affects the chance of crossing successfully .A crossing attempt may either be deliberately decided upon or may happen by accident, for example, by being swept away while gathering aquatic resources. Deliberate crossing decisions occur because the hominins would expect valuable resources at the opposite shore.Within Scenarios A and C, we modeled a basic water movement behavior which is either randomly directed (Scenario A) or is directed towards the target shore (Scenario C). Our results suggest that paddling and swimming should be considered as an option for crossing water barriers of an extent of up to and including 10 km.Homo erectus or Homo floresiensis, possessed the biomechanical requirements for swimming [The extent of the hominin s’ paddling or swimming performance is not known and is the subject of current research . Even thswimming , this apswimming , 76, 77.It is well known that, at present, humans can perform long distance swimming, even in open waters, for example, in the context of competitions , 79. An We wish to note that in our model, the agents do not vary in swimming skill. However, there may be substantial differences in the performance between individuals with regard to prior training, technique and physical fitness . IncreasAs we do not know the extent of hominin swimming performance, we tested a variety of performance levels. By simulating two different basic water movement performances in Scenarios A and C, we showed that distances of several kilometers would not prevent hominin agents from crossing, even without the use of rafts. Instead, the behavioral difference in the swimming intention has a tremendous impact on the crossing success. The crossing of water barriers should, therefore, be considered as an option in hominin route selection and should be ruled out only if a variety of behavioral models have been tested and failed in crossing the barrier.Analogous to the paddling and swimming performance, we modeled two different levels of maritime transportation devices within the Scenarios B and D. Scenario B represents random movement on water with a drifting carrier, while the majority of the hominin’s body remains under water. In Scenario D, we assume the hominins to remain atop a raft and are in control of the direction .Our results suggest substantial differences in the crossing successes between using a drifting carrier (default CSR ~ 0.04) and rafting directedly (CSR ~ 0.63).For Scenario D, the hominins would have required knowledge on how to build basic rafts, for example, from bamboo, bundles of grass or other plant fibers. This would have required them to be able to fasten bundles or sticks together in order to build a floating surface sufficient in size to carry a single person or a group across water. Building such a compound raft would require some degree of collaboration. Moreover, some form of steering technology, such as sticks or paddles is required. Directed rafting probably goes beyond the cognitive skills presumed for hominins during Out-of-Africa 1, but may function as a reference scenario for Middle or Late Pleistocene hominins.Actualistic experiments in which simple rafts were built with natural materials, such as bamboo, illustrate that distances of up to 80 km could be travelled . When foTherefore, these experiments should be complemented by computer simulations; these would allow simulating attempts of crossing sea straits under various environmental scenarios that resemble conditions during the Pleistocene. By designing appropriate scenarios, our model permits the testing of the whole range of maritime navigation that is presently considered for hominins, ranging from random and passive drifting to directed and controlled rafting. Furthermore, our model permits the design of specific scenarios for other methods of water crossings, such as directed rafting with the body in the water or hitch riding on the back of elephants , as wellThe scenarios presented in this study represent two different types of movement in water which may be grouped, one the one hand, into scenarios of movement in random directions (Scenarios A and B) and, on the other, scenarios of movement directed towards the target area on the opposite shore (Scenarios C and D).Given that Scenarios C and D consistently resulted in considerable higher CSRs than Scenarios A and B, this suggests that the directionality of movement has a major effect on the success of crossing a water barrier.Furthermore, swimming directly towards the target area (Scenario C) outperformed drifting in a random direction (Scenario B). This indicates that the use of a drifting carrier may not be advantageous if the movement is not directed and that the reduced energy loss due to drifting bears no advantage, at least within favorable conditions and distances of up to and including 10 km. Hence, the use of rafting technology would not be a requirement for crossing such types of water barriers. It is assumed that rafting becomes more important particularly for longer distances exceeding 20–30 km . A more The comparatively shorter time span that one spends in the water while moving directly towards the target area explains the higher success rate in crossing the water barrier because the chances of dying by hypothermia, dehydration or exhaustion is reduced accordingly. Due to the reduced exposure to dangers from the water, successful crossings of larger water barriers become more likely.Thus, when applying the model, assessing the capabilities of the modeled hominins in performing directed movement towards the target area needs to be carefully considered in order to apply an appropriate skill of movement in the water.Strong currents may, however, constrain the extent to which a directed movement towards the target area is possible. On the one hand, currents may be utilized as an assisting force. as is observed with fish migration –94 or byAlthough our results suggest that moving directly across the water barrier is substantially more successful in establishing a founder group on the opposite shore, crossings by moving randomly in the water also occurred. We note that, in the real world, the chance of crossing a water barrier accidentally to form a founding population is very low, due to the low probability that individuals can accumulate in sufficient numbers within a short period of time, e.g., months or years.We may suggest that moving directedly is not a necessity for crossing a water barrier, because if given enough attempts, random movements will lead to reaching the target area sooner or later. Such unintentional water crossings are documented for Iguanidae , lizardsNonetheless, moving randomly in water without orientation may only occur on rare occasions. These could be, for example, in the context of natural catastrophes, such as tsunamis or hurricanes, or in the water under bad weather conditions, such as fog, high waves or during the dark. Therefore, the simulation runs for the Scenarios A and B, where the agents in the water move in random direction, may overrepresent these rare occasions.Consequently, when the model is applied for a particular case-study, the chances of environmental conditions which induce loss of orientation need to be assessed and considered.We note that accidental crossings may also lead to directed crossings. For instance, stranded hominins may emit cues, such as smoke from campfires, or returnees might tell of distant lands. This would, of course, require the ability to interpret indirect signs and communication. For this purpose, the model could be adapted by combining undirected and directed scenarios, as well as integrating complex social interactions, such as collaboration, shared use of rafts and communication. This could be realized by defining groups of individuals as an agent and by allowing agents to exchange information with each other. The same applies for cognitive skills such as monitoring, learning and planning, which permit more complex decisions, such as the cost-benefit assessment of the crossing or the deliberate use of currents. Although agents do not adapt during a simulation run, the behavioral scenarios represent different stages of maritime adaptations, whereat paddling is the lowest stage and rafting the highest. Hence, adaptation rates and their effects on crossing success could be tested by switching between behavioral scenarios during a simulation run. In addition, aquatic resources can be added to the model by assigning resource qualities to the water patches. Agents who are adapted to an aquatic environment and specifically seek aquatic resources will spend more time on the coast, opening up more opportunities for crossing attempts.The success in arriving at the opposite shore is determined by the ability to survive the crossing and the establishment of a viable founder population. In the model, the crossing attempts may fail because agents leave the map or die in the water. When examining the causes of disappearance, the majority of agents disappear by leaving the map on land by moving towards the border of the map, either at the source or the target area. The former case refers to an absence of crossing, whereas the latter case refers to a failed crossing because the agents have already left the target area before a founder population could establish itself. Although some agents may disappear by being washed away by currents, the majority of agents die of hypothermia in Scenarios A-C see . FurtherThe causes of disappearance for all scenarios, are given in the As agents may leave the map at the borders, individuals need to accumulate in sufficiently high numbers in the target area before too many of them have left the map. Our simulation experiments show, that when the crossing is directed, such founder populations may establish several times within the time frame of three months under favorable conditions. Although less likely, founder groups may also establish by accident, when crossings are undirected. We may assume that such accidental crossings become more likely when considering extended periods of time, e.g., years or generations, instead of only three months. Populations may establish after several years or generations  with preHominin expansions are considered as processes that occurred across time periods of millions of years and across continents. Therefore, these are often modeled as large-scale diffusion models , 22, 23.Our agent-based model allows one to simulate water crossing with hominin agents that are capable of paddling, drifting, swimming or rafting across a water barrier embedded in artificial landscapes with random environmental variables. As we intended to explore the general effects, we tested five factors in conjunction with four scenarios of skilled performances of movement in water. From the simulations, we can identify circumstances under which water crossing could have been more, or less, likely. We found that the directionality of the movement behavior of the agents along with their correspondingly wide perception have a substantial effect in performing successful and sustainable crossings of sea straits.This model can act as a template for the simulation and quantifying of the crossing of other landscape features that are assumed to be barriers, such as deserts and/or mountain ranges.The strength of our model is that it represents the process of water crossing dynamically, by the hominin agents, transitioning from land into water and moving through the water by paddling, drifting, swimming or rafting within rather small spatio-temporal scales. Therefore, future research should intensify the exploration of hominin behavior and develop frameworks with which behavioral models can be integrated and subsequently tested in simulations. Our agent-based model allows formalizing and explicating the numerous conceptual models for hominin sea crossing from archaeology and paleoanthropology e.g. , 11–14),–14,11–14The output variable CSR, with which we quantified the success of the crossing, helps to identify the extent to which environmental behavioral factors contribute to the crossing success. Furthermore, the CSR can be used as an interface, to integrate the results of small-scale processes into large-scale “diffusion-like” hominin expansion models and draws a bridge to the otherwise disconnected scales.Our model could be applied to reconstructed maps that match with paleoanthropological questions. Moreover, our model can be adapted to provide CSRs for various time periods as well as geographical contexts, for example, hominin expansions out of Africa . WhetherOur model can improve existing large-scale agent-based models for hominin sea crossing during the Middle- to Late Pleistocene  or moderOur studies show that the model introduced here is responsibly calibrated, thoroughly tested and ready to be applied in geographically explicit analyses. When applied, the simulations will provide valuable insights into the role of sea crossings during hominin expansions.S1 FigThe bars show the average CSRs from 27 simulation runs for each factor level.(TIF)Click here for additional data file.S2 FigThe bars show the average CSRs from 27 simulation runs for each factor level.(TIF)Click here for additional data file.S3 FigThe bars show the average CSRs from 27 simulation runs for each factor level.(TIF)Click here for additional data file.S4 FigThe bars show the average CSRs from 27 simulation runs for each factor level.(TIF)Click here for additional data file.S5 FigThe bars show the average CSRs from 27 simulation runs for each factor level.(TIF)Click here for additional data file.S1 FileThe bars show the average number of deaths from 27 simulation runs for each factor level. Agents may disappear by leaving the map or when dying.(PDF)Click here for additional data file.S2 FileCSV output of the simulation experiments presented in this paper and Python scripts used for creating the graphs.(ZIP)Click here for additional data file.S3 FileSimulation settings of all simulation runs that were conducted in this study with their respective Figure references.(XLSX)Click here for additional data file.


The monitoring of electrical equipment and power grid systems is very essential and important for power transmission and distribution. It has great significances for predicting faults based on monitoring a long sequence in advance, so as to ensure the safe operation of the power system. Many studies such as recurrent neural network (RNN) and long short-term memory (LSTM) network have shown an outstanding ability in increasing the prediction accuracy. However, there still exist some limitations preventing those methods from predicting long time-series sequences in real-world applications. To address these issues, a data-driven method using an improved stacked-Informer network is proposed, and it is used for electrical line trip faults sequence prediction in this paper. This method constructs a stacked-Informer network to extract underlying features of long sequence time-series data well, and combines the gradient centralized (GC) technology with the optimizer to replace the previously used Adam optimizer in the original Informer network. It has a superior generalization ability and faster training efficiency. Data sequences used for the experimental validation are collected from the wind and solar hybrid substation located in Zhangjiakou city, China. The experimental results and concrete analysis prove that the presented method can improve fault sequence prediction accuracy and achieve fast training in real scenarios. Reliability and stability are the most important aspects to guaranteeing the safe operation of electrical networks and power systems. Early and accurate fault prediction is a valuable and urgent subject in electrical equipment maintenance and power grid operation. An electrical line trip fault usually happens in a power grid system ,2; it wiIn the past decades, many studies have been proposed in the power system faults prediction area ,12,13,14(1)The long time-series line trip fault prediction method using the improved stacked-Informer network is adopted; it exploits more comprehensive temporal information of long sequence input measurement data, includes normal and abnormal current and voltage data of power lines, and then predicts the short sequence output fault. This method achieves a superior generalization performance.(2)The strategy of gradient centralization (GC) technology is introduced and embedded into the optimizer, replacing the original Adam optimizer usually used in the DNN model for a GC+Adam optimization. GC can be viewed as a projected gradient descent method with a constrained loss function; it operates directly on gradients by centralizing the gradient vectors in order to have zero means. This technology improves the training time of the long sequence time-series fault prediction markedly and improves the accuracy and efficiency of the presented methodology.(3)Real measurement long sequence data is collected by electrical sensors at a wind solar hybrid power station, which is located in Zhangjiakou City, China. The recorded long sequence data is divided into two datasets, and both of them are conducted on the proposed methodology in order to prove a superior performance in real-scenario application.The rest of this paper is organized as follows. The related work is described in Many real-world applications require a long sequence time-series forecasting (LSTF) with a high prediction accuracy, and it has great significances for electrical fault sequence prediction in power systems; this prediction will provide an early alarm to ensure a normal and safe power grid operation. Some models, such as RNN, LSTM and Transformer , are gooIn order to deal with this problem, an efficient LSTF transformer-based model named Informer was firstly proposed by Zhou et al.  in 2021,token  and the padding element of 0 is X0; the concatenated feature map and attention composition are fused to predict the long sequential output instantly (yellow series). The Informer network contains two main parts: encoder and decoder. In Line trip faults are usually existing electrical faults in the power system because of aging, bad insulation and other reasons. It is dangerous to cause massive power blackouts. In general, the recorded electrical measurement data collected by the wind-solar hybrid power station includes the current and voltage during the operating process, and it reflects the occurring faults according to some abnormal fluctuation of data.In this subsection, the architecture of the proposed methodology is introduced in detail. The presented method is an improved stacked structure based on the Informer network. Besides, the GC technique is introduced in the optimization instead of the previously used Adam optimizer. The optimization technique is very important to effectively and efficiently train a DNN model. Different from most of the adopted optimizers, such as SGD and Adam used in DNN, a new GC technique is embedded into the Adam of the presented methodology. GC technology was firstly proposed by Yong et al. in 2020 ; it dealw whose gradient is M represents the dimension, and L is the objective function. The GC formulation in Equation (1) is very simple and efficient, only the mean of the column vectors of the weight matrix is computed, and the mean is removed from each column vector to accomplish the GC technology. We can also rewrite a matrix form of Equation (1) to be Equation (2):W is the weight matrix, P is the projection matrix in the hyperplane with a normal vector in weight space, and Suppose the gradient is obtained by back-propagation; then, for a weight vector gradient . After o in Ref.  , Intel Xeon E5-2650 v3 @2.3GHz CPU, and 128GB RAM. The operating system is Windows 10, and the deep learning framework is PyTorch.For the long sequence time-series line trip fault prediction, the major task is to forecast whether there are faults or not during the operation of a power system. The experimental validations of the improved GC+Adam optimizer and original Adam optimizer are discussed in this subsection, respectively. In order to evaluate the performance of the different optimizers, the base architecture selects the original Informer network and the proposed improved stacked-Informer network for an extensive comparison, respectively. All of the parameters are listed in In this subsection, we select the original Informer network and improved stacked-Informer network as the base architecture to evaluate the performance of the improved GC+Adam optimizer and Adam optimizer, respectively. Comparative results are shown in To better explore the superior characteristic of the presented method using the improved stacked-Informer network with the ADM + GC optimizer, it is compared with many other original Informer networks with different optimizers, respectively. For the final fault sequence prediction result, In order to further validate the performance of the long time-series fault sequence prediction, we give the evaluated MSE and MAE results in To increase the operational reliability and stability in power systems, a data-driven method for long sequence time-series line trip fault prediction using an improved stacked-Informer network with GC optimization is proposed in this paper. First, two real datasets of three-phase current and voltage values were recorded by electrical sensors during normal operation and a brief occurrence of a line trip fault. The long sequence time-series data is fed into the proposed methodology for the output fault sequence prediction. The stacked structure of the Informer model can improve the generalization ability of the fault prediction, and GC technology increases the training efficiency of the presented method. The experiments were conducted on real data collected from the wind-solar hybrid power substation in Zhangjiakou City, China. Specifically, the corresponding experimental results prove the increasing generalization performance ability. All in all, the improvement of the proposed methodology is noteworthy when compared to the current novel Informer model. In prospective work, we will further explore the performance of different stacked structures of the network and do more comparisons with other outstanding fault prediction methods.