P < 0.05). For patients with orotracheal intubation, using a new oral fluid suction device to fix the tracheal intubation can effectively prevent the displacement of the tracheal intubation, protect the oral mucosa and lips, and reduce the pain of the patient. At the same time, the oral fluid suction device designed in this study has low production cost, strong practicability, and is suitable for clinical promotion.Tracheal intubation is an important access to general anesthesia surgery or respiratory support in critically ill patients. Orotracheal intubation is the most common method of establishing artificial airways in clinical practice. Tracheal intubation and mechanically assisted breathing are among the important steps in the clinical rescue of critically ill patients. During tracheal intubation, it easily causes iatrogenic skin damage, unclear mouth, ulcer, and other oral mucosal complications due to the improper fixation method and excellent dental pads. Therefore, the purpose of this study was to design a novel oral suction device for patients with orotracheal intubation and explore its safety, convenience, and comfort in the clinical application of orotracheal intubation patients. From October 2016 to April 2017, a total of 232 patients with mechanical ventilation through orotracheal intubation in the Department of Surgery and Critical Care Medicine were selected by the convenience sampling method. According to the random number table method, 232 cases were divided into the experimental group and control group, with 116 cases in each group. The experimental group used a self-designed oral fluid suction device to fix the tracheal intubation; the control group used the traditional method, placing ordinary disposable tooth pads, and then using 3M tape to fix the tracheal intubation. The incidence of oral mucosa and lip pressure ulcers, patient comfort, and tracheal tube displacement were observed and compared between the two groups. The incidence of oral mucosa and lip pressure ulcers in the observation group using the self-designed oral fluid suction device to fix the tracheal intubation was significantly lower than that in the control group. At the same time, the comfort of the patients was significantly higher than that of the control group, the incidence of tracheal tube displacement was significantly lower than that of the control group, and the differences between the above indicators were statistically significant ( Tracheal intubation is an important channel for breathing support for patients undergoing general anesthesia surgery or critically ill patients. Orotracheal intubation is the most commonly used method for establishing artificial airways in clinical practice. Tracheal intubation and mechanically assisted breathing are one of the important steps in the clinical rescue of critically ill patients . During P > 0.05), and the two groups of patients were comparable.232 patients with mechanical ventilation through orotracheal intubation in the Department of Surgery and Critical Care Medicine from October 2016 to April 2017 were chosen as the research subjects. Inclusion criteria were as follows: age ≥ 18 years old; first intubation after admission; all patients underwent orotracheal intubation for mechanical ventilation; no coagulation dysfunction; no patients with pulmonary infection; no oral disease; days of orotracheal intubation ≥ 2 days; tracheal intubation; and the skin around the mouth, cheeks, and neck before the tube was intact without ulceration. Exclusion criteria were as follows: patients with severe oral diseases; patients with severe bleeding and coagulation disorders; oral infection; those with respiratory tract infection and pulmonary infection before tracheal intubation; those who do not actively cooperate with the researcher; and those with mental illness and communication disorders. The investigation of this study was carried out with the consent of nurses and patients and signed a voluntary informed consent form. The study protocol has been reviewed and approved by the hospital ethics committee. According to the random number table method, 232 patients were divided into the experimental group and the control group, with 116 cases in each group. General information is given in The oral liquid suction device includes a guide tube and a liquid suction body. The side wall of the top is inwardly recessed to form a circular groove. The liquid suction body is cylindrical, and its diameter gradually decreases from the top to the bottom. The main body is used to absorb oral fluid, which is made of multiple layers of gauze wound in a spiral manner. The adjacent gauze layers of each layer are stacked in sequence. The liquid absorbing body includes a first body, a second body, a third body, and a fourth body. They are sequentially stacked from the inner ring to the outer ring. The liquid body is formed by winding a piece of gauze with flushing seam covered by adjacent layers of gauze. One end of the guide tube is inserted into the suction body from the top of the suction body; the other one is exposed to the suction body. One that is exposed to the suction body is used for connecting with the negative pressure suction device. The side wall of the guide tube inserted in the suction body has several guide holes. While oral liquid in the suction body flows into the guide tube through the guide holes, the suction body is passed through the guide tube and the oral fluid in the duct is discharged. The diversion tube includes a first diversion tube and a second diversion tube. One end of the first diversion tube is connected with one end of the second diversion tube. The first diversion tube is located in the suction body inside and the second is located outside. The outer wall of the end connected with the second guide tube is recessed inward to form a ring-shaped slot, and the second guide tube is connected to the second guide tube. The inner wall of the connected end of a guide tube protrudes to form a ring-shaped clamping table, and the ring-shaped clamping table is clamped into the circular clamping groove and can rotate in the circular clamping groove. A sealing ring is installed between the ring-shaped clamping table and the ring-shaped clamping groove. Production materials are as follows: 4 pieces of self-sealing gauze 8 cm ∗ 30 cm, 1 piece of 4 mm (12F)  ∗ 550 mm suction tube, 1 piece of 3M sterilized breathable film 6 cm ∗ 7 cm, and a pair of scissors. Made according to the method above, the product is a one-time use, as shown in When the patients have been successfully intubated by the orotracheal tube, the control group placed ordinary disposable dental pads between the upper and lower teeth of patients. The experimental group used an oral fluid suction device . Placed between the upper and lower teeth, the dental pad and the tracheal tube were tied side by side with tape and then fixed on the patient's cheeks with 3M tape [The occurrence of pressure sores on the lips was evaluated before each oral care. According to the staging of oral ulcers, stage 1 pressure ulcers: flaky or streak-like bruising on the skin around the lips and gums; stage II pressure ulcers: the skin around the lips and gums is purple-red, with blisters and superficial mucosal ulceration; and stage III pressure ulcer: A superficial ulcer stage, with full-thickness skin destruction, which can penetrate deep into the subcutaneous tissue and deep tissue .Patients were assessed by uniformly trained nurses. According to the patient's self-perception, a visual analog scale (VAS) was used, with 0 indicating comfort, 5 indicating general comfort, and 10 indicating very uncomfortable. Higher scores indicate lower comfort levels.(1) No displacement: the graduation of the catheter tip from the incisors was completely unchanged. (2) Mild displacement: the distance between the tip of the catheter and the scale of the incisor teeth is within 0.5 cm. (3) Moderate displacement: the distance between the tip of the catheter and the scale of the incisors is 0.5–0.8 cm up and down, but it does not cause the catheter to fall off or slide down to one main bronchus. (4) Severe displacement: the tube is dislodged or the tube slides down and blocks one side bronchus.χ ± SD). Comparisons of materials between two groups were performed by the t-test. The enumeration data were expressed as percentages, the comparison of the enumeration data was performed by the chi-square test, and the comparison of the rank data between groups was performed by the Wilcoxon rank-sum test. Differences between groups were considered statistically significant at P < 0.05.SPSS 19.0 was used to analyze the data, the quantitative materials fit a normal distribution. The quantitative data are presented by (P < 0.05), as given in There were no severe oral mucosal and oral pressure ulcers in both groups. The incidence of oral mucosa and lip pressure ulcers in the observation group after intubation was significantly lower than that in the control group, and the difference was statistically significant , as given in The comfort of the patients in the experimental group after intubation was significantly better than that in the control group, and the difference was statistically significant (P < 0.05), as given in Severe displacement did not happen in both groups of patients. The degree of displacement in the experimental group is significantly smaller than in the control group. The difference was statistically significant (Tracheal intubation is the most basic rescue technique for critically ill patients. It is of great significance in the maintenance of airway patency, ventilation and oxygen supply, airway suction, and prevention of aspiration . OrotracIn actual work, the material of ordinary disposable dental pads is hard, and its flanks are easy to compress the skin of the patient's lips, resulting in damage to the skin there. For patients with missing dentition, the dental pad is in direct contact with the gums, and prolonged pressure will damage the gums. In addition, the back of the tooth pad is narrow, and there is general physical friction between the lingual surface and the upper jaw. These physical frictions can easily lead to damage to the oral mucosa and easily lead to oral complications. The oral liquid suction device designed in this study has a cylindrical body with no side wings, which reduces the friction between the device and the lips, oral mucosa, teeth, and gums. At the same time, the material of the device is gauze, which acts as a buffer between the tooth pad and the gums for patients with missing dentition, which greatly reduces the hardness and reduces pressure on the teeth and gums.In the traditional method, the common dental pads are mostly straight-shaped wings, which are made of a very hard plastic product, which is easy to compress the lips and gums, causing discomfort. The main material of the newly designed oral fluid suction device in this study is gauze, which changes the original plastic shell structure column, avoids pressure on the perioral skin, and improves the comfort of the patient. In traditional operations, nurses need to use a suction tube to suck the secretions in the mouth of patients with endotracheal intubation through the tooth pad. Such manipulation can stimulate nausea and even vomiting in the patient. After adopting the newly designed oral liquid suction device, the end of the guide tube is directly connected with the negative pressure suction device, which reduces the stimulation of the patient's oropharynx and increases the patient's comfort.In critically ill patients in the intensive care unit, due to endotracheal intubation and matching tooth pads, the patient's mouth cannot be closed voluntarily, and the swallowing function is limited. This situation can easily lead to difficulties in excretion of oral secretions and changes in saliva properties. Furthermore, under the influence of various factors, the oral environment of patients may even be disturbed, bacterial reproduction will increase, and oral cleanliness will be significantly reduced . The aveThe oral fluid suction device provided by the utility model has the advantages of simple structure and convenient use, which solves the problem that the oral fluid of the patient cannot be easily sucked out and increases the operation difficulty of the medical staff. The oral fluid suction device can not only quickly suck out the oral fluid and avoid pressure sores on the patient's lips and damage to the oral mucosa but also ensure the comfort of the patient, and at the same time, it is convenient for medical staff to use the oral fluid suction device, and the work efficiency is improved. In addition, the oral liquid suction device has low manufacturing cost, strong practicability, and is suitable for promotion.


The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block’s pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row. In recent years, visual-based control strategies have successfully spread in a wide variety of robotics contexts . NowadayThe game of Jenga is a perfect example of a challenging benchmark for robotic perception and control. In recent years, researchers have tackled the game with disparate platforms and approaches, adopting sophisticated manipulators ,18 and cA DL-based instance segmentation model fine-tuned on a custom Jenga tower dataset realized in simulation, which effectively allows the system to segment and select single blocks;An eye-in-hand visual control strategy that carefully handles the block extraction;A 1D force sensor to correctly evaluate the removability of a specific block.This work presents a cost-effective system to play Jenga with the educational robotic manipulator—e.DO by Comau and a custom pushing finger as an end-effector. Our proposed solution is based on the combination of visual and tactile perception to handle the human-level complexity and the high precision required by the task. In particular, compared to previous attempts to play Jenga with a manipulator, we adopted a single RGB-D camera and a basic 1D force sensor as complementary hardware to the robot arm, considerably reducing the cost and complexity of the solution. An illustrative sequence of frames depicting our robotic system in action is shown in Our extensive experimentation validates all the sub-components of the proposed system. First, we study the force reaction on the 1D sensor and obtain a threshold-based decision policy to classify the extraction of the block as feasible or not. Then, we provide details on the training and testing of the instance segmentation model, comparing results obtained on simulated and real-world images. Moreover, we test our visual perception pipeline composed of segmentation and pose estimation of a group of blocks, measuring the tracking time during several runs. The adopted visual servoing control strategy is tested on the two major behavioral features of interest: precision, in terms of distance between the point of contact and the center of the block, and efficiency, estimated as the time required to align the end-effector to a block. Finally, we evaluate the overall performance of our solution by counting consecutive successful block extractions.The paper is organized as follows. In In this section, we first describe previous studies aimed at playing Jenga with a robot. Then we introduce the computer vision task of instance segmentation and its state-of-the-art and report similar works adopting visual servoing and multisensory control in robotic applications.Jenga is a common benchmark for robotic systems, allowing for a direct comparison of methods, experiments, and results. So far, few works have proposed a complete visual-based robotic system to play the game autonomously. At the same time, several studies focus on a partial aspect of the game with a specific solution.Recently and most notably, Fazeli et al.  delved iKroger et al. , who adoWang et al.  propose A two-fingers, anthropomorphic, 7-DOF industrial robot arm is used in . An eye-A fine-grained kinematic analysis of the physics behind weight, friction interactions, and stability of the Jenga tower is studied in , both duDifferently, Yoshikawa et al.  investigSimilar to most related studies, our solution uses a force sensor to detect push failures and empirically check the removability of blocks. On the other hand, our perception system presents several novelties: a block identification approach based on an instance segmentation neural network, an eye-in-hand camera configuration, and a visual servoing control for the manipulator.During the last few years, deep learning  has achiSuppose a more fine-grained semantic description is needed. In that case, the object detection task has the objective of localizing instances that belong to target classes with the regression of bounding boxes. This detection allows the system to understand the scene hierarchically depicted in the image, assign multiple labels, and spatially identify the objects in the image reference frame. Popular methodologies for object detection ,37,38,39On the other hand, the semantic segmentation task aims at assigning a semantic label at the pixel level by predicting masks that identify the portion of the image belonging to a certain class. Classical approaches to this task are based on fully convolutional networks organized in an encoder–decoder fashion ,41, whicInstance segmentation aims to localize single instances by predicting masks. This approach allows the most precise interpretation of the input image since it avoids coarse bounding box localization by identifying masks at a pixel level. Several methods have been proposed in the literature to solve this task. mask-RCNN  extends Industrial robotic tasks, such as assembly, welding, and painting represent standard scenarios where manipulators execute repeatable point-to-point motion by using off-line trajectories . HoweverVisual-based control strategies recently emerged as valid candidates for real-time trajectory computation and correction. In particular, visual servoing was introduced in 1979  and refeVisual servoing taxonomy distinguishes two approaches  accordinOn the other hand, position-based visual servoing leverages a priori geometrical information on the object to derive the corresponding visual features. Recently, hybrid schemes have tried combining the two techniques and leveraging 2D and 3D visual features. In , an aeriIn this section, we frame the Jenga game and translate the rules of the game into a methodological set of requirements for the robotic system. First, a player’s final goal is safely removing blocks from the tower. In the original game setting, a player has to place each extracted block at the top of the tower to validate its round and continue with a new block extraction. In our robotic experimental setting, we remove this rule and aim to extract as many blocks as possible from the tower without reallocation. This choice is mainly related to the limited workspace of the e.DO manipulator, designed for educational purposes, since it cannot reach the fallen blocks behind the tower. Moreover, our custom end-effector, similar to a human finger, cannot perform grasping and instead extracts blocks by pushing. As the first practical task, a Jenga player should be able to select one of the blocks of the tower to be extracted. To this end, each block of the tower is identified in our system using an instance segmentation deep neural network. Moreover, we define a heuristic block selection policy based on the idea that extracting multiple blocks from the same tower level is not recommended. Moreover, as better detailed in The identification and pose estimation of each block of the tower is the first step to selecting a suitable piece and approaching it;A removable block cannot be identified only by visual analysis: the integration of a tactile perception system is needed;A sufficiently precise alignment between the end-effector and the center of a target block allows the arm to extract it successfully by simply pushing.Therefore, our solution is based on the following assumptions:A complete illustrative schematic of the system proposed to play Jenga is depicted in The proposed methodology’s first step relies on identifying the blocks present in the tower by visual analysis. We adopt a deep learning-based method to perform instance segmentation. The necessity of using a deep learning approach is caused by the fact that wooden blocks do not have easily-distinguishable features while having predictable positions in the camera frame instead. In this setup, the vision system can benefit from the ability of neural networks to generalize to different light conditions and points of view. Moreover, training the model on an exhaustive dataset can make it robust towards missing blocks and tower misplacement during the game. Among different approaches for image semantic analysis, instance segmentation aims at detecting all the objects of interest in an image at the pixel level. Given an input image of the tower, the model should output a set of possible Jenga block candidates, together with their respective pixel masks. Instance segmentation can therefore be seen as a combination of an object detection task with a mask prediction task. By translating the position of the blocks from the image reference frame to the robot reference frame, it is possible to achieve block localization.a as possible Jenga block candidates. Each anchor prediction consists of the class confidence c, four bounding box regressors, and k mask coefficients. These mask coefficients are used to weigh the mask prototypes produced by the second branch. Both branches are based on convolutional layers applied to the features extracted by the common backbone.We select the YOLACT++  architecc score to produce the actual output. Moreover, as in standard object detection algorithms [The two branches are finally followed by a mask assembly block, which combines the masks with the coefficients predicted by each anchor to reach the final instances prediction. At inference time, anchor predictions are thresholded with a certain value gorithms ,37, an NOur solution defines a heuristic block selection policy based on physical and empirical considerations. Visual information cannot provide sufficient information to determine the status of a block. The imperceptible tolerances of each block cause minuscule variations in pressure that prevent visual-based systems from understanding which blocks are truly removable. The blocks in the higher layers are easier to extract, i.e., they can be pushed out of the tower by applying a smaller force. However, pushing from a decentralized contact point contributes to the formation of torques on the block that causes asymmetry in the tower. This effect is amplified on upper layers due to lower friction forces and may affect the stability of either adjacent blocks or the tower itself. Instead, the friction force increases for blocks located at lower levels, making the extraction harder and risking the Jenga tower falling.Such considerations imply the need for a policy to select the block to extract. However, the policy must also consider the physical dimensions of our 6DOF anthropomorphic manipulator. Indeed, during the extraction primitive, the robot has to be parallel to the block, which implies a loss of DOFs. These orientation constraints restrict the robot’s workspace, so the manipulator can only reach a limited range of tower levels. In order to overcome these issues, the policy divides the entire tower into two subspaces according to the robot’s workspace. In particular, the subspaces correspond to the upper and lower levels, each with a predetermined number of tower levels. By convention, the numeration of tower levels carries in ascending order, where one corresponds to the lowest level. In addition to this vertical division, the policy is also initialized with the block direction for each level. Indeed, two possible tower orientations have a relative rotation of As human players usually do, we initially adopt a random policy that selects the Jenga pieces in one of the subspaces. Then, the manipulator approaches the chosen block and starts pushing. At this point, force sensor data are collected to evaluate the block’s status according to the adopted force threshold, as explained in The policy repeats the process until all levels contained in the subspace are tested. After that, the system changes the subspace to test additional levels until the tower collapses. More in detail, selecting the higher layers as the starting subspace leads to three subspaces in total, with extra layers being included in the final subspace. Alternatively, if the process begins by extracting blocks from the lower layers, there will be only two subspaces, with extra layers being included in the high subspace. Predicted block masks are the input of a post-processing unit implemented using OpenCV. It estimates the position and orientation of the desired block to be extracted with respect to the camera. This information is the first requirement to approach the block’s face starting the manipulator’s tracking and visual servo control. The pose estimation is performed with an intrinsic calibrated camera through the Perspective n Points (PnP) algorithm optimized for planar points . PnP is Therefore, the PnP algorithm provides an initial estimate of the 6DoF pose of the group of blocks (target and adjacent blocks) using segmentation mask corners to initialize the model-based tracker.The geometric dimensions of a standard piece of Jenga  control  can be cIn this regard, a robust tracking algorithm is a fundamental and challenging component of visual servoing control, which is required to guarantee smooth trajectories and allow a faster convergence to the target. As described in the previous sections, the segmentation masks are used to estimate the position and orientation of the desired block. At this stage, the tracking system receives both the initial pose estimation and the segmentation masks and combines them with a 3D model of a Jenga block. Therefore, it detects the target to establish a continuous mapping of the 6DOF pose of the Jenga block in the camera field of view. Specifically, the adopted stereo model-based tracking method ViSP  combinesHowever, as mentioned in the previous Moreover, the visual features of the group of blocks of interest are collected from different perspectives before the tracking is started. This acquisition phase enables recovery tracking in case of sudden movements by exploiting the acquired pairings and re-initializing the block pose.Z and s, defined as Our approach adopts the eye-in-hand visual-servo configuration, whose extrinsic camera parameters are estimated based on the end-effector design. Thus, we indicate with v represents the linear and angular camera velocities that are computed through the following control law:The Tactile perception is a fundamental aspect of a human Jenga player since identifying potential removable blocks is not possible via visual analysis only. According to this, we decide to incorporate a low-cost 1D force sensor in our system, which only provides information about the perpendicular reaction force between the end-effector and the target block. Hence, while previous works adopted a 6D force sensor to adjust the direction of the end-effector while pushing , we takeA challenging aspect of the game is that the more push attempts are performed, the more unstable the tower becomes, as extracted blocks or rotations perturb its structure due to pushes and retractions. Therefore, two thresholds detect immobile or moving blocks in oretical  and empiThe experimental setup includes the anthropomorphic educational manipulator, e.DO manufactured by Comau, a depth camera Intel RealSense D435i, a MicroForce FMA piezoresistive force sensor , an Arduino Nano 33 BLE, and 3D printed components such as a rotating base, an end-effector design extension, and camera support. The software runs on a single computer with an i7-8750H CPU and 16 GB of RAM. Thus, one of our goals is to investigate and prove the effectiveness of state-of-the-art with low-cost equipment. Visual control adopts an eye-in-hand configuration with camera support, while the small force sensor is placed on top of the end-effector extension connected to the Arduino Nano.In this section, we first describe the experiments carried out to define the static force threshold used to check the removability of blocks. The real Jenga blocks present small differences in dimensions, generating a diverse pattern of friction forces in the tower each time it is rearranged for a new game. Although  tried toThe measurements are taken with a complete tower configuration during the block extraction, starting the data collection of force reaction from when the contact between the block and end-effector starts and the force sensor detects a non-zero value. The experiment is run multiple times on different tower levels, mixing the blocks’ disposition each time to test random friction conditions. The plot in The choice of the threshold force values is not trivial, as it affects the system’s ability to classify the block’s removability state. By setting higher threshold values, more blocks can be classified as false positives, i.e., they are perceived as free to move, whereas they cannot be safely extracted. On the other hand, imposing lower threshold values affects system performance leading the system to avoid feasible block extractions.As depicted in In this section, we report the method and the details of the procedure to train the instance segmentation model with a carefully devised synthetic dataset, as well as the results obtained by experimental validation. The experimentation aims to assess the quality of the deep learning model and its generalization to the real pictures of our experimental environment.The materials are randomly sampled and assigned to all 48 cuboids to change their look;A total of 6 to 24 cuboids are randomly displaced along their x and z axes by a distance between −4 mm and 4 mm;A point light source is positioned by randomly sampling a height of 0.1 ÷ 0.5 m spanning a circular arc of A total of 2 to 9 random blocks are removed from the tower to create holes;Camera position is sampled on a circular arc of The training of the instance segmentation model is entirely performed on a synthetic dataset crafted from a 3D model of the Jenga tower. Using the 3D modeling software Blender and its Python APIs through BlenderProc , hundredThe camera is placed to capture two faces of the tower at the same time. For each configuration, 10 samples are acquired with the full tower and 10 with random missing blocks. The procedure is repeated for each random scene obtaining a diverse and complete dataset with different light conditions, block displacements, missing blocks, camera angles, and views of the experimental conditions. The scene’s background is then filled with black, white, or gray. The render time for all the 880 images was 4 hours on an i7-9700K CPU. The test set is composed of 20 real manually-annotated pictures from our experimental setup for a total of around 800 segmented tower blocks. The images are taken in slightly different light conditions and camera positions.The input 640 × 480 images are rescaled to 550 × 550 to be compatible with network input requirements. We adopt a ResNet-50 backbone with pre-trained weights on ImageNet. We perform 8000 training iterations (69 epochs) with a batch size of 8, SGD optimizer with a momentum of 0.9, and a weight decay of As the main metric to assess the quality of the instance segmentation, we adopt the widely used average precision at different values of intersection-over-union  to extract the visual features and compute the The experiment begins with the robot in a default pose at the same height as the tower’s first level. Then, a timer starts and automatically stops when the end-effector reaches the desired pose with a fixed tolerance on the visual servo error magnitude as The second part of the test aims to assess the manipulator’s accuracy in pushing blocks. It consists in manually stopping the robot as soon as the end-effector reaches the target and measuring the distance between the contact point and the center of the block. Defining Finally, the different modules are integrated, and tests of the architecture are performed on the full system. The experiment includes 18 trials following the semi-automatic loop explained in A mobile block is correctly pushed without perturbing the stability of the tower;The binary classification model correctly identifies the block as immobile, and the arm retracts without causing the tower to fall.After each attempt, the policy updates the block status obtained from visual and tactile measurements during the extraction process. In particular, force sensor data are used to detect the block status, while visual measurements provide feedback on the result of the extraction primitive. The latter integrates pose estimation information and the direct kinematics of the manipulator in order to verify the complete extraction of the block from the tower. The minimization of the angular tracking error in the visual servo law allows positioning the end-effector as parallel as possible to the block by pushing it out of the tower , placing the extracted blocks on top to form a new level, and aborting extractions when failures occur.The robot reaches a singularity position when the target block level is at the edge of the workspace;The tracking is lost as either the tower is in bad light conditions or the target block lies outside the field of view;The initial pose estimation is not precise enough because the segmentation model does not detect all the corners of the block correctly.Minor failures do not stop the game, as they do not cause the tower to collapse. The operator can abort an attempt when the system fails for reasons unrelated to the extraction. In this case, the loop continues on the next iteration, and the policy is not updated. Possible minor failures are:To minimize these failures and prevent the perturbation of the tower’s stability, we make some conservative choices. Different force thresholds are assigned depending on the game’s phase, i.e., which subspace of blocks is currently tackled. This way, the policy addresses the first subspace with an aggressive force threshold of 0.32 N. After all the first subspace levels have been tested, the policy addresses the second subspace with a cautious threshold of 0.18 N. The effect of adding the extracted blocks on top is twofold: their weight contributes to changing the friction between the blocks and shifting the center of gravity of the tower. Furthermore, each new level is considered in the policy for additional attempts. This way, we obtain a minimum of 42 attempts (3 per level) and 14 block extractions (1 per level), plus 3 additional attempts and 1 extraction for every newly formed level.The results are reported in Each experiment ends when the tower falls. In our experiments, the fall occurred after 13.8 correct attempts on average . In most experiments, the fall was caused by the increasing instability of the tower after multiple extractions. However, a few experiments ended earlier because of poor tracker performance in providing the position and orientation during the push movement or incorrect detection of mobile blocks (two experiments ended after only 5 extraction attempts). The highest registered score counts 14 successful extractions and 11 detected unmovable blocks. The results are comparable to those of , which sIn this paper, we propose a complete system to play the challenging game of Jenga with cost-efficient robotic hardware. The contribution of this work goes beyond the Jenga game, proposing an advanced, adaptable solution for accurate manipulator control in delicate robotic tasks. We demonstrate that a visual-based approach for perception and control can provide the robot with significant benefits in terms of scene understanding and control accuracy.Differently from previous works, the main components of our system are a deep instance segmentation neural network used to identify each block of the Jenga tower and a visual tracking and control pipeline to continuously adjust the pose of the end-effector as it approaches the block. A low-cost 1D force sensor is integrated into the system to check the removability of the target block, drastically reducing the cost and complexity of the overall system. Our extensive experimentation shows the remarkable performance of each fundamental component of the solution. After examining the accuracy and stability of the perception and control units, we evaluate the whole system by playing Jenga and reaching a maximum of 14 successive block extractions. Future works may see the advancement of the reasoning capability of the robot: reinforcement learning agents can be investigated to optimize the planning policy for the game. Moreover, a visual-based sensorimotor agent could eventually replace the controller by directly mapping segmented and depth images to velocity commands for the end-effector.


Tinnitus is a hearing disorder that is characterized by the perception of sounds in the absence of an external source. Currently, there is no pharmaceutical cure for tinnitus, however, multiple therapies and interventions have been developed that improve or control associated distress and anxiety. We propose a new Artificial Intelligence (AI) algorithm as a digital prognostic health system that models electroencephalographic (EEG) data in order to predict patients’ responses to tinnitus therapies. The EEG data was collected from patients prior to treatment and 3-months following a sound-based therapy. Feature selection techniques were utilised to identify predictive EEG variables with the best accuracy. The patients’ EEG features from both the frequency and functional connectivity domains were entered as inputs that carry knowledge extracted from EEG into AI algorithms for training and predicting therapy outcomes. The AI models differentiated the patients’ outcomes into either therapy responder or non-responder, as defined by their Tinnitus Functional Index (TFI) scores, with accuracies ranging from 98%–100%. Our findings demonstrate the potential use of AI, including deep learning, for predicting therapy outcomes in tinnitus. The research suggests an optimal configuration of the EEG sensors that are involved in measuring brain functional changes in response to tinnitus treatments. It identified which EEG electrodes are the most informative sensors and how the EEG frequency and functional connectivity can better classify patients into the responder and non-responder groups. This has potential for real-time monitoring of patient therapy outcomes at home. Tinnitus is characterized by the perception of sound in the absence of an external sound source. Although the effects of tinnitus vary from person to person, most patients suffer from varying degrees of fatigue, stress, sleep problems, difficult in concentrating, memory loss, anxiety, and irritability . The mosThe acute effect is injury to the cochlea and auditory nerve, following auditory plasticity, tinnitus-related effects centralise over time. Due to the heterogeneity of tinnitus, treatment response varies. Clinical judgements about the likelihood of treatment response are currently difficult, and patients are often switched through several treatments before finding an effective intervention. AI may help in personalising the prescription approach to tinnitus by indicating those interventions most likely to be effective for any given patient. This would improve clinical efficiency, reducing patient distress and optimising benefits. Although several AI methods have been applied to brain data in order to classify (diagnosis) tinnitus  ,4,5,6, tThe current study focuses is on EEG, which allows for high temporal-resolution recording of cortical electrical activity via electrodes positioned over the scalp ,14. By uIn this paper we explain the methodology of our predictive models as well as the feature selection methods used for improving the performance of the models. We report the predicted results of treatment outcome classification . A comparative analysis using different feature selection methods is reported on EEG data in the frequency and functional connectivity domains.This research aims to develop an AI-based system for predicting the change in patients’ severity of tinnitus over a period of treatment using computational models of brain data collected prior to and following a tinnitus treatment (the EEG data is explained in ). The prThe clinical impact of the proposed methodology is applying AI for the early prediction of response to tinnitus treatment when only baseline data is used. This would allow for an optimal selection of treatment options for patients through the prediction of response to each treatment. We also identified a subset of EEG sensors as informative features that increase the prediction accuracy and make the further development of wearable diagnosis and prognosis AI tools more effective. Sensors have seldom been used clinically in relation to sensory disorders. The application of sensor technology to tinnitus is novel, and the algorithm and its application are innovative approaches to this complex sensory disorder. The literature has clearly identified the need for biomarkers, and this is amongst the first studies looking towards effective wearable solutions in our field.We used two datasets  from patients with tinnitus  from 64 BioSemi active Ag/AgCl recording electrodes. Electrode locations corresponded to the extended international 10/20 system. Electrodes were attached to a fitted BioSemi head cap. Parker Signa gel was applied at each electrode site to ensure reliable conductivity between electrode and scalp. Continuous EEG signals were recorded on a Dell Optiplex 7040 desktop computer at an 8192 Hz sample rate with a 64-channel BioSemi ActiveTwo system referenced to the common mode sense active electrode and grounded to the driven right leg passive electrode. The EEG signals were down-sampled to 256 Hz. The temporal lengths of these EEG signals were segmented into multiple intervals of 256 time-points . This resulted in 6642 EEG samples, which were used for training deep neural networks to predict the patient’s tinnitus treatment outcomes (class labels: responder vs. non-responder defined with respect to the changes in their TFI score after 3 months of treatment). The class labels of the EEG samples correspond to the TFI changes in the behavioural data of the same patients.Tinnitus severity numerical scales (TSNS) have been widely used to assess tinnitus severity and have demonstrated good test–retest reliability and concordance with other participative measures of tinnitus. Participants were asked how much a problem their tinnitus was (0 not a problem−5 very big problem). Numeric rating scales were used to measure tinnitus perception along five dimensions: how strong, intrusive, uncomfortable, unpleasant the tinnitus signal was, and how easy it was to ignore the tinnitus signal  . The litDepression, Anxiety, and Scale (DASS)  is a valIn the current research, we applied neural networks for the classification of EEG data (only baseline EEG) in order to predict patients’ treatment outcomes (responder and non-responder groups) labelled after 3 months of tinnitus sound treatment. To this end, a tinnitus avatar was first generated that was identical to the individual with tinnitus. This sound was then morphed slowly over time until it was identical to an environmental sound. Within the field of virtual reality, one study has attempted to synthesize an auditory replica of tinnitus. However, this relied purely on auditory thresholds, tinnitus pitch, and participant preference judgments .In the current study, the computational AI model has the potential to distinguish the baseline EEG patterns of patients who are likely to respond to the tinnitus treatment over time. In order to identify different types of predictive patterns from EEG data, we considered two forms of EEG-driven information as inputs to the neural networks: (1) frequency domain  was applied to transform each of the EEG samples into the frequency domain image. As described in Equation (1) and Reference , we firsncy band . The ampData from the sequence of 1 s epochs was used as a temporal input into the neural networks for training. In this experiment, because the EEG datasets were presented in the form of images, we employed a CNN in deep learning , commonlfication , image sfication ,38, and fication . In the xyP represents the cross-spectral density of the signal x and y. Parameters xxP and yyP show the power spectral density. The generated EEG functional connectivity graphs were used as inputs to train a deep neural network. Hereafter, we employed MLP as the classifier.In this section, EEG data was transformed into functional connectivity representing the relationship between different cortical regions over time. Graph theory was applied to each EEG frequency domain in order to convert it into a graph-based representation of the frequency domain, shown in As explained in For the prediction of tinnitus treatment outcomes, only the baseline EEG data from patients were used to train the CNN model. The class label information for the baseline EEG data was defined according to the patients’ TFI scores after treatment. The trained CNN model was then tested using patients’ baseline EEG data, which were excluded from the training to predict whether the patients are likely to be classified as non-responders or responders. For the training and testing of the CNN, the EEG samples were split into training and test datasets in a ratio of 8:2, meaning that there were 5314 training samples and 1328 test samples. As reported in k-fold cross validation for training and testing the CNN model. The results of different numbers of fold validation are reported in To assure that the CNN model was robust and stable, we also applied In our experiment, the raw EEG signals were transformed into functional connectivity graphs with respect to the coherence measured between every pair of EEG channels as defined in Equation (1). The functional connectivity is a dynamic graph in which the nodes are the EEG channels, and the arcs demonstrate the squared coherence between the channels. The graph was updated with respect to the changes in EEG signal over time, thereby, generating time series information that can be used for the training of deep learning neural networks based on MLP.We applied MLP as the model to predict classes (treatment outcomes) based on the EEG functional connectivity graphs, each with a size of 64*64 cells (representing pairwise correlations between 64 EEG channels). We applied 8:2 training and testing split for the classification reported in M is the number of patients, i is the i-th channel, and j is the j-th patient. j in EEG channel i. j in the same EEG channel i. Here the range of I is [1–64] and j is [1–8].The experiments for the prediction of tinnitus treatment were conducted using all 64 EEG channels. In order to reduce the dimensionality of the data and the computational model complexity, those EEG channels that were not significant for tinnitus diagnosis were detected and eliminated. We applied a variety feature selection method to determine the most important EEG variables affected by tinnitus treatment. Here, we developed a new approach, called greatest change channel selection (GCCS), for measuring the importance of the EEG channels in classifying the EEG samples to pre- and post-treatment states. This feature-selection method focused on finding the channels with the greatest changes caused by treatment. Equation (3) calculates For the GCCS feature selection method, we calculated the changes in the amplitude of each EEG channel before and after treatment. After calculating the average degree of treatment effect p-values as the criteria for ranking the importance of the channels. The larger the p-values, the higher the importance of the corresponding channel. This was calculated by measuring the correlation between the before-matrix data and the after-matrix data based on r-regression. Then, the cross correlation was converted to an F-score and then to a p-value.For the FR method, the model returned the F-statistic and The RF method ranked the importance of channels for each tree according to the impurity, which was calculated based on variance.The ET method ranked the importance of the channels according to the impurity-based feature importance. The higher the rank, the more important the channel. The importance of a channel was computed as the  total changes in the value of the channels.The RFE method determined the importance of each channel through the feature importance attribute returned by the model. Then, the least-important channels were removed from the current set of channels. This step was repeated recursively on the channel set until the required number of channels was finally reached.AI diagnostic prediction is an extremely promising new field for the treatment of tinnitus. Due to the richness and variety of treatment methods for tinnitus and the variation in tinnitus treatment outcomes amongst individuals, the use of AI diagnosis and prediction may lead to a better treatment plan. This shortens the treatment time and allows patients to receive more targeted and personalized treatment. The goal of this study was to build artificial intelligence models to predict the outcome of tinnitus treatment. This research applied different neural networks (including CNN and MLP) to learn from patients’ EEG data for predicting their treatment outcome. The neural networks modelled EEG frequency features and functional connectivity features, and they resulted in up to 99% accuracy of for prediction of patients who were responders or non-responders to treatment. To the best of our knowledge, no other method for predicting the treatment outcome of tinnitus by analysing frequency features and functional connectivity has been published. Moreover, there is a lack of EEG studies investigating tinnitus treatment. Compared with previous works using time-domain features as the object of analysis ,41,42,43So far, the highest accuracy of our predictions comes from the frequency-domain model, at 99.52%. At the same time, the functional connectivity-based model also performed very well, with an accuracy of 99.41%. In the process of model training, we used 70% of the data for training and the remaining 30% for network performance testing. We also conducted 5-fold cross-validation on the model, and the results were all above 90% accuracy. This suggest that our model is sufficiently robust. This is strong evidence that EEG signal analysis by AI models can reliably predict the outcome of tinnitus treatment. In addition, we applied five conventional feature-selection techniques  to identify the top predictive EEG variables that lead to increasing the prediction accuracy. We also proposed a new feature-selection approach, called GCCS, which resulted in the best prediction accuracy when compared to other methods. The GCCS method identified FC3, P8, P4, T8, and CP5 as the main predictive EEG variables.This research also enabled visualisation of the topography of the human brain for frequency-domain features of the EEG, and visualisation of the functional connectivity of the human brain. The frequency-domain visualization  allowed The research proved that within EEG data, both frequency and functional connectivity contain significant information showing brain changes as a response to treatment. Each of the frequency images and the functional connectivity graphs was generated using a 1 s EEG signal (256 time points), and they were used as inputs to deep learning and achieved a greater accuracy of prediction compared to using the whole time points of raw EEG signals.For the future, a real-time prognostic digital health system is planned to be developed based on a small number of EEG variables (selected through feature selection) for the potential design of a wearable system for patients as home. To this end, a more robust AI model needs to be trained using more EEG data from tinnitus patients. This will allow us to identify a generalised group of EEG channels associated with tinnitus and the effect of treatment.


The remaining useful life (RUL) prediction is important for improving the safety, supportability, maintainability, and reliability of modern industrial equipment. The traditional data-driven rolling bearing RUL prediction methods require a substantial amount of prior knowledge to extract degraded features. A large number of recurrent neural networks (RNNs) have been applied to RUL, but their shortcomings of long-term dependence and inability to remember long-term historical information can result in low RUL prediction accuracy. To address this limitation, this paper proposes an RUL prediction method based on adaptive shrinkage processing and a temporal convolutional network (TCN). In the proposed method, instead of performing the feature extraction to preprocess the original data, the multi-channel data are directly used as an input of a prediction network. In addition, an adaptive shrinkage processing sub-network is designed to allocate the parameters of the soft-thresholding function adaptively to reduce noise-related information amount while retaining useful features. Therefore, compared with the existing RUL prediction methods, the proposed method can more accurately describe RUL based on the original historical data. Through experiments on a PHM2012 rolling bearing data set, a XJTU-SY data set and comparison with different methods, the predicted mean absolute error (MAE) is reduced by 52% at most, and the root mean square error (RMSE) is reduced by 64% at most. The experimental results show that the proposed adaptive shrinkage processing method, combined with the TCN model, can predict the RUL accurately and has a high application value. With the rapid development of computing methods and information technology, modern production systems have become more complex . In receIn the past decade, RUL prediction technology has made great progress, mainly including model-based methods, data-driven methods and hybrid methods ,4. AmongDeep learning has a strong nonlinear mapping ability and feature extraction ability, and it is increasingly used in the field of RUL prediction and health monitoring . In RUL The temporal convolutional networks (TCNs) have been the latest improvement in the CNN structure, which extracts historical data using the dilated causal convolution (DCC). The dilated causal convolution usually includes fewer layers than the classical CNN but can capture the same receptive field. Therefore, TCNs have a better time series prediction ability than CNNs . In addiTo solve these problems, this paper proposes a RUL prediction method based on adaptive shrinkage processing and temporal convolution network (AS-TCN). Firstly, the vibration signals monitored by multiple channels are directly used as the input of the prediction network, without prior knowledge to extract features. Secondly, in AS-TCN, TCN residual connection and dilated causal convolution are used to extract long-term historical information, and an adaptive shrinkage processing sub-network is introduced to adaptively eliminate different noises. Finally, the PHM2012 bearing data set and XJTU-SY bearing data set are compared with the three most advanced methods, respectively. The average MAE is reduced by 52% at most, and the average RMSE is reduced by 64%, which verifies the effectiveness of the proposed method.The main contributions of this paper can be summarized as follows:(1) A new framework of RUL prediction based on AS-TCN is proposed. It can directly use multi-channel monitored data as network input, without prior knowledge to extract features and effectively capture the key degradation information of bearings, thus realizing the end-to-end prediction process.(2) Using a TCN network to build the main network can help it remember a large amount of complete historical information, avoid the shortcomings of long-term dependence and improve the accuracy of RUL prediction.(3) Add an adaptive shrinkage processing subnet to the TCN block. The sub-network can adaptively adjust the threshold of the soft threshold function to minimize the redundant information related to noise while retaining the features that can better reflect the degradation information.The rest of this article is organized as follows. Causal convolution was first used in the WaveNets model  to learnthe past . Since there is no recurrent connection in casual convolution, a parallel input of time series data can be adopted, so causal convolution has a faster training speed than RNNs, particularly for the large sample time series . HoweverThe DCC architecture with a kernel size of The TCN uses residual learning to simplify deep network training. He et al.  proposedBatch normalization (BN)  has usuaShrinkage processing refers to soft thresholding, which is a function that shrinks input data in the direction of zero to retain negative or positive characteristics and sets the characteristics approaching zero to zeros. In this way, it has been proven that useful information can be well preserved and noise related features can be eliminated . The sofTo detect the degradation information of equipment comprehensively, this study uses the operation-to-failure data collected by different sensors as a network training dataset. However, due to environmental impact, changes in operating conditions, performance degradation and other factors, the noise level also changes. Therefore, when the signal samples are converted into the feature map through the stack layer, it is necessary to customize the threshold of the feature map. To this end, an adaptive shrinkage training subnet is constructed in the TCN framework. This subnet adaptively adjusts threshold The working mechanism of the adaptive shrinkage processing is illustrated in The AS-TCN block consists of DCC and an adaptive shrinking sub-network. After a series of operations, the input data, which represent a picture with redundant information and degradation features, are input to the TCN block, and different thresholds are obtained by the adaptive shrinkage subnetwork for different features. Next, the soft threshold function is used to eliminate redundant information to retain degenerate features. Moreover, the TCN block uses an identity path to reduce the difficulty in model training. The internal details of the DCC sub-network stacked with custom layers are presented in The RUL prediction process is illustrated in The trained network model is used to perform RUL prediction on the test set. The simulation results are expressed according to the fit between the RUL predicted value curve of the test set and the true value curve, and the absolute error (MAE) and mean square root error (RMSE) are used as indicators to evaluate the prediction effect.The structure of the RUL prediction network based on the AS-TCN designed in this paper is shown in Network parameters include trainable parameters (such as the weights and offsets of convolution kernels) and untrained super parameters (such as the number and size of convolution kernels). The super parameter needs to be set in advance, and its influence on the network prediction effect and training time needs to be comprehensively considered. After repeated experiments and comparisons, the final super parameter setting is shown in The proposed prognostic method was validated using two accelerated rolling bearing degradation test datasets. The bearing operation-to-failure dataset  released® Xeon E5 2696 v2 @ 2.5 GHz, the memory was 128 GB and the GPU was a Nvidia® GeForce 3070Ti (8 GB); the operating system was Microsoft® Windows 10 (64-bit); and the programming language was PythonTM3.9 based on the Tensor Flow-GPU 2.6.0 deep learning framework.The server configuration used in the laboratory was as follows: the processor was an IntelDifferent operation settings may lead to different sensor values, and the obtained data represent different physical characteristics. Therefore, in order to eliminate the impact of data irregularities on the prediction effect, data normalization is carried out before the model is trained and tested, and the minimum and maximum values of the data set are converted to the range of  to improve the calculation speed of the model. The calculation formula is as follows:To extract the degradation features from the network model better, the horizontal and vertical vibration signals were used as the network input data, 0.1 s of were divided into a sample  CNN. The conventional CNN model had no DCC and adaptive shrinkage processing mechanism.(2) AS-CNN. In this model, compared to the AS-TCN, only the TCN was replaced with the traditional CNN, while the rest of the network parameters were kept the same. (3) TCN. The AS-TCN is converted to TCN. The AS subnet is removed, and other structural parameters remain unchanged. This model is used to evaluate the effectiveness of the AS subnet. (4) DSCN . A new d(5) CNN BiGRU .The mostWe used cross validation to deeply evaluate the performance of AS-TCN and other methods in RUL prediction. We divided the 14 data sets listed in The experimental results are shown in In order to more intuitively represent the MAE and RMSE of different methods, the results in The XJTU-SY dataset was provided by Xi’an Jiaotong University and Changxing Sumiao Technology Company . The tesAs in Experiment Case I, we used cross validation to deeply evaluate the RUL prediction performance of AS-TCN and other methods. The data set division and experimental results are shown in The results in In the process involved in the prognosis and health management framework, the prediction of remaining useful life (RUL) is a key task of PHM and is also the basis for decision-making on management activities. Therefore, this paper proposes a RUL prediction model for AS-TCN equipment to predict the RUL of rolling bearings. In addition, ablation experiments and comparisons with state-of-the-art models are conducted, and two experimental cases are provided to verify the superiority of the proposed prediction method to the other methods.Based on the results, the following conclusions can be obtained:(1) Adding a DCC hierarchy with a residual block to the TCN module can be used to seize longer-time collection records.(2) The training subnet adopts an adaptive mechanism to learn the soft-thresholding function adaptively so as to reduce the information related to noise, retain degenerated features and extract the health status information of equipment.(3) The validity of the proposed RUL prediction method based on the AS-TCN is verified by applying the rolling bearing to the test bench; the proposed method is compared with several different methods. The experimental results show that the proposed AS-TCN has excellent RUL prediction ability and thus has an important reference value for the actual remaining life prediction.In a future study, we will consider the degradation characteristics of the same bearing under different working conditions, realize the RUL prediction of bearings under different working conditions and adaptively divide the health state and degradation state mechanism to achieve more accurate RUL prediction.


Hemodynamic factors are believed to be closely related to IA growth. However, the underlying pathophysiological mechanism that induces the growth sequence in tandem intracranial aneurysms (IAs) remains unclear.This study involved five patients with tandem IAs. Aneurysm models were reconstructed based on image datasets. A novel vascular restoration algorithm was proposed to generate the hypothetical geometry of the healthy parent vessel before each IA formation in the concatenated structure. Detailed hemodynamic patterns and morphological features were revealed under various growth sequences of tandem IAs to investigate the flow-driven mechanism of IA growth. Potential hemodynamic indicators of IA formation were proposed.The patient cases were divided into two groups based on the size difference of tandem IAs. In the group with a similar size of tandem IAs, the position of the vortex core was associated with the site of the secondary aneurysm, while in the group with a significant size difference of the IAs, the position with the maximum curvature of the parent vessel plays a significant role in aneurysm formation.This study preliminarily revealed key hemodynamic and morphological indicators that determine the formation of tandem IAs. The proposed vascular restoration algorithm that provided the pre-aneurysm vasculature might be useful in investigating the flow-driven mechanism of IA growth, thus contributing to the risk evaluation of secondary aneurysm formation. Intracranial aneurysms (IAs) are injuries or diseases of the intracranial arterial wall, resulting in localized or diffuse expansion or bulging of the wall. IAs occur in around 2–3%  of the gTo our knowledge, because of the low rupture rate  and the Computational fluid dynamics (CFD) can provide detailed hemodynamic information for the assessment of IAs and has become increasingly popular in IA research , 13. AmoVortices were reported to alternate intravascular flow directionality on a broad scale and may lead to a pathological vasculature change, which is correlated with endothelial cell alignment , 19 and This study included five patients with tandem IA, with separated IAs and the parent vessel's skeleton. We thoroughly investigated the possible preferential initiation of an aneurysm by using important hemodynamic parameters. Ultimately, the characteristics that might contribute to subsequent IA formation were summarized. To the best of our knowledge, this is the first study to determine the potential hemodynamic indicators that can indicate how a single aneurysm triggers the growth of the secondary one, which can enrich the results of previous studies and help surgeons identify risk factors that may cause tandem aneurysms.This study was approved by the Ethics Committee at Beijing Tiantan Hospital. This study involved five patients with tandem IAs. For each patient, CTA sequences were obtained from the clinic and imported into the Mimics Innovation suite for 3D segmentation. Mimics reconstructed vessel geometry using the image segmentation method based on signal intensity. The vascular restoration algorithm was used to create the pre-aneurysm vasculature. First, the original model, the skeleton, and the inflow and outflow cross-sections in the adjacent aneurysm area without diameter mutations were extracted. Next, the semi-automated algorithm was used to generate a cylindrical tube with a uniform diameter changing between the inflow and outflow sections. Three models were reconstructed: Model P, Model D, and Model A. One representative example of the original model and the results of 3 sets of pre-aneurysm vasculatures are shown in Pa·s and a mass density of 1,066 kg/m3. The vessel walls were assumed to be rigid walls with a non-slip boundary condition. The inlet and outlet boundary conditions were extracted from @neufuse software, where the database of human blood vessel boundary function information was contained. After simulating three cardiac cycles with 100 time steps each, 100 equally spaced time points were saved for data processing and analysis from the last cardiac cycle. The flow governing the Naviér-Strokes equation was solved by ANSYS CFX 19.2  based on unstructured mesh.The reconstructed aneurysm models were meshed in ICEM CFD and consisted of the tetrahedral grid in the core region and the prismatic grid near the vessel wall. Blood was simulated as an incompressible and Newtonian fluid with a dynamic viscosity of 0.0035 Q>0, where positive Q indicates that the rotational flow occupies the main position in the flow domain  is the spatial wall shear stress gradient vector, S is the symmetric rate-of-strain tensor, and Ω is the antisymmetric rate-of-rotation tensor.The above hemodynamic parameter expressions are shown in To quantitatively evaluate the changes in hemodynamic parameters between different areas containing IA inception, areas that contained aneurysm occurrence of each model were chosen based on the regional positioning algorithm. First, the artificially selected aneurysm neck plane was used as the positioning plane, and the pre-aneurysm vasculatures were projected onto the neck plane to determine the area of the aneurysm inception. Next, the IA inception region was localized in the partial area according to the principle that the angle between points in the aneurysm neck curve and each point in the IA inception region is equal to 360°. These points were imported into the Geomagic Studio 2012 (64-bit) software and converted into surfaces. One representative example of the ROI region in three sets of pre-aneurysm vasculature is shown in To further investigate the possible causes of IA initiation, the morphological parameters that could reflect the degree of curvature of the models, such as curvature and DM, were extracted and analyzed. Curvature is a parameter of the degree of the curve deviating from the straight line; DM is the ratio of the straight-line distance to the curve distance between the start point and the endpoint. The skeletons of pre-aneurysm vasculatures were extracted from all patients to extract morphological parameters. Additionally, the comparison of morphological parameters to different groups of the models can be seen in Considering the correlation between curvature and model specificity, we performed a secondary analysis of the maximum curvature position of all patients. Hemodynamic calculations were performed on all pre-aneurysm vasculatures, and hemodynamic parameters are drawn in AFI and GON are two WSS-derived parameters invented to explore the risk of aneurysm formation. As shown in Combined with previous studies, we considered negative AFI with a higher absolute value, positive GON, and TAWSS with a higher value to be risk factors inducing initiation of secondary IA , 17, 26.The vortex core is an important manifestation of the vortex flow, which could offer a key vehicle for describing an aneurysmal flow field in the respects such as location. The vortex core was extracted by Q-criterion with a given threshold value of Q, which could provide a direct description of the vortex. This study set the same threshold to extract the vortex core for each case. The visualization results of the vortex core of a typical case are displayed in The possible preferential initiation model in the left graph appeared to have a limited number of cores. The structures of the intra-vascular vortex core were shaped like a small tube, gathering near Surface A in In this study, the variation of hemodynamic indices in the IA initiation regions was inconsistent with different patient models. Preliminary, limited hemodynamic parameters were included in our study to ensure the accuracy of estimating a possible IA growth priority model. In this study, we attempted to evaluate the flow complexity inside the vessel by adding an additional hemodynamic indicator vorticity core to provide a unique perspective for the study of aneurysm initiation. Collectively, this study provided ideas for evaluating tandem aneurysms' inception and treatment by analyzing hemodynamic factors.The TAWSS, GON, and AFI were gathered as a parameter group and quantified the changing percent in the IA inception region under two different growth sequences. The adoption regulation of these parameters limited the indices into parameter groups because they have been extensively researched and proved to be correlated with the IA inception region and could impact IA initiation , 15, 17.Q and λ2 values to condense the distributions vortex cores region  . In thisDue to the scarcity of the parental vascular model before aneurysm growth, the vascular restoration method occupied our research's main position. Sunderland et al. proposed a method for restoring the vessels' lumen by VMTK in 2019 . AlthougThere were several limitations to this study. First, the patient types are similar, so a selection bias may exist. Second, the patient-specific velocity waveforms could not be obtained from the hospital. To simulate the pulsating CFD, we used the database of human blood vessel boundary function information instead. Although the vortex core was proved to be the judgment factor for second aneurysm growth in our research, future research needs to expand the sample size and enrich the sample. Finally, although the simple Q-criterion was adopted in our research, there are many other definitions of the vortex, including the λ criterion and so on . In futuIn our research, the possible preferential initiation models were estimated by changing hemodynamic parameters. The ROI of IA initiation was located to improve the accuracy of the dataset. The growth of one single aneurysm significantly affects the hemodynamics of another aneurysm, suggesting a possible IA growth priority model for tandem aneurysms. Ultimately, obvious vortex cores were extracted in the IA incidence region in the same size group. Although similar vortex core information was not found in the different size groups, the point of maximum curvature was found to correspond with the area of the IA incidence. The vortex core information might guide tandem aneurysms to induce the growth of the subsequent aneurysm. The order of temporal occurrence for multiple IAs and vortex cores may provide a new perspective to studying aneurysm rupture. The assumption that smaller aneurysms have a higher risk of rupture and an unstable artery wall was proposed because of the pathological conditions triggering degenerative remodeling for a long time.The original contributions presented in the study are included in the article/The studies involving human participants were reviewed and approved by Ethics Committee at Beijing Tiantan Hospital. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.DC and AL designed the research studies. YY, YM, FY, and YS collected the patient data and analyzed the data. YY and XT wrote the manuscript. All authors contributed to the article and approved the submitted version.This work was supported by Beijing Natural Science Foundation (Z190014 and L192010).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


A wireless vision sensor network (WVSN) is built by using multiple image sensors connected wirelessly to a central server node performing video analysis, ultimately automating different tasks such as video surveillance. In such applications, a large deployment of sensors in the same way as Internet-of-Things (IoT) devices is required, leading to extreme requirements in terms of sensor cost, communication bandwidth and power consumption. To achieve the best possible trade-off, we propose in this paper a new concept that attempts to achieve image compression and early image recognition leading to lower bandwidth and smart image processing integrated at the sensing node. A WVSN implementation is proposed to save power consumption and bandwidth utilization by processing only part of the acquired image at the sensor node. A convolutional neural network is deployed at the central server node for the purpose of progressive image recognition. The proposed implementation is capable of achieving an average recognition accuracy of 88% with an average confidence probability of 83% for five subimages, while minimizing the overall power consumption at the sensor node as well as the bandwidth utilization between the sensor node and the central server node by 43% and 86%, respectively, compared to the traditional sensor node. In recent years, the idea of Internet-of-Things (IoT), which was once a mere ambitious thought, has become a reality. At its core, IoT is all about a worldwide network of unique and identifiable objects that communicate using standard protocols . RealizaOne such field that has seen significant progress is the field of image sensors. The advancements made in this field spewed out what is referred to as wireless vision sensor networks (WVSNs). The WVSN system consists of multiple vision sensor nodes (VSNs)—often a smart camera; a central server node or base station (BS); and in some instances, an aggregation node . The reaIn most applications, the vision sensor node is expected to perform some vision processing tasks, in what represents a marked progression from traditional vision sensors. This, however, has added new design constraints in terms of power consumption and bandwidth, among others . The desIn the architecture presented in The complicated trade-off between processing power consumption and bandwidth utilization is not trivial, and this has necessitated the emergence of a third architecture that strives to strike a balance between the two illustrated in Unlike the mainstream image compression algorithms, designing compression algorithms for WVSNs requires careful design considerations ,20. The Under these design guidelines, the Microshift algorithm, which is one of the most recent and effective compression algorithms for WVSNs, was proposed in . This coInspired by the features of the Microshift algorithm, we propose a WVSN implementation that is capable of achieving progressive image recognition. Firstly, the processing power consumption at the sensor node and transmission bandwidth are minimized by progressively compressing and transmitting five out of the nine subimages. Secondly, a convolution neural network (CNN) is deployed at the central server node to perform progressive image recognition on the subimages as they arrive. Last but not least, we present the trade-off necessary between recognition accuracy and expenditure in terms of overall power and bandwidth utilization.The rest of this paper is organized as follows: In this section, the details of the proposed WVSN implementation are presented. The system as a whole can be divided into two processing stages, namely the sensor node processing stage and the central server node processing stage, as illustrated in The processing capability of this stage is limited and, therefore, few processes are implemented. As illustrated in i’.A The I is the captured image, and M is the matrix that contains replication of the Microshift pattern that covers the full captured image.The micro-shifted image is now quantized with a coarse quantizer of smaller resolution  to generate the sub-quantized micro-shifted image The Microshift compression algorithm reported in  is an efThe sub-quantized micro-shifted image, For the progressive compression, the first subimage For the remaining subimages, they are compressed using “inter-prediction”. This technique uses the information from the previously compressed subimage. This means that the pixels in the second subimage will be predicted based on the pixels from the first subimage, while the third subimage is predicted using information from the first and second subimages, and so on.This process is termed “Progressive Compression” because as soon as a subimage is compressed, the compressed bitstream is transmitted wirelessly to the central server node.To improve the compression ratio, the Microshift algorithm utilizes a further lossless encoding step. The implementation steps are summarized below:For WVSNs, this process reduces the power consumption at the vision sensor node as the full image is not compressed in its entirety at once. Furthermore, the progressive transmission of bits reduces the bandwidth demand since bits of the subimages are sent sequentially as opposed to sending the compressed bits of the full image.The available processing power available at this stage is quite large compared to the sensor node processing stage. As shown in The proposed system is designed around the progressive decompression capability of WVSNs. As earlier stated, the compressed bits are transmitted from the sensor node to the central server node in the sequential order of subimages. As these compressed bits are received at the central server node, the respective subimages are reconstructed using the “FAST Decompression Algorithm” presented in .The FAST Decompression algorithm merits its name from its speed of execution. It is inspired by the “Heuristic Decompression” in . This teLet us assume that a pixel given by .(2)⋓i=[For each of the neighboring pixels in the patch containing To implement the FAST decompression algorithm progressively is quite easy. For any subimage The progressive decompression method provides improvement in reconstructed subimages as more bits are received. An illustration of the progressive Decompression using the “Elaine” standard test image is shown in A convolutional neural network (CNN) is deployed at the central server node for image recognition. CNNs are a replication of the human brain for machine learning applications and have been integral components in the history and development of deep learning . InsteadThere are many pre-trained CNNs available. These include GoogleNet, VGGNet, ALexNet and Resnet. In this work, the pre-trained CNN is based on the ResNet50 base model illustrated in The weights of the ResNet50 are frozen and the output layer is replaced with a new layer. The new output layer will be retrained on the Caltech101  dataset The Caltech101 dataset contains around 8677 images divided randomly into 101 classes. The data is partitioned with 80% used for neural network training and 20% for testing.The experiment was conducted on the progressively decompressed Caltech101 dataset that is generated in this work. Each image was converted into nine progressively decompressed subimages and the recognition algorithm was performed on each one.In this section, the proposed WVSN implementation is analyzed to verify its performance in terms of power consumption and bandwidth utilization.The proposed WVSN implementation is capable of progressive image recognition. Recognition in the proposed work was completed progressively using the subimages. Typically, the decoder at the central server node waits for all the subimages to arrive, recombines them, decompresses the recombined image, and finally runs it through the recognition neural network, as illustrated in In this experiment, we observed how recognition accuracy trends relative to the increase in number of received subimages. Transfer learning was employed for the purpose of training. The weights of the model’s convolutional layers were frozen or unchanged, while the output layer of the ResNet50 model was removed and replaced with a new output layer. The network was then trained on the Caltech101 testing dataset and reported an accuracy of 94.9%.For image compression, we employed the Microshift algorithm that is capable of achieving, on average, a compression ratio of around 5.6–5.7 (approximately 1.3 bits per pixel). We simulated the central server node by using progressive decompression and recognition.The average progressive recognition accuracy for the generated testing dataset with increasing number of received subimages is presented in The observed trend from To further highlight the effectiveness of the progressive recognition, we present in From To validate the benefits of the implemented WVSN, a performance comparison is presented in terms of hardware resource utilization, bandwidth utilization and overall power consumption.In From Assuming a WiFi transmission channel between the sensor node and the central server node that is capable of transmitting data at 40 Mbps at 0.00525 μW/bit , we compFor the assumed WiFi channel transmitting at 40 Mbps, the time required to transmit one bit evaluates to 25 ns. From To validate the benefits of any WVSN implementation, the overall power consumption needs to be minimized when compared to the traditional sensor node that simply acquires and transmits the image to the central server node. The overall power consumption (Equation .(7)PT,wIn the traditional sensor node, the overall power consumption (Equation .(8)PT,tIt should be stated that the acquisition power consumption is similar for both our WVSN implementation and the traditional sensor node, i.e., quations  and 8),,Pacq,w=PFor our WVSN implementation that needs to send a total of 7681 bytes  across tIn Equation  and 11)11), it iFor our WVSN implementation, the full design flow was accomplished. First, the design was completed in Verilog, generated by MATLAB HDL Coder, then synthesized using the TSMC 0.18 In this paper, we have proposed a wireless vision sensor network implementation that is capable of early and progressive image recognition. The proposed implementation works on the subimages transmitted by the sensor node and progressively runs them through a recognition algorithm, as they are decompressed at the central server node. While maintaining a high level of accuracy with significant confidence level, the progressive processing of images helps to save power and transmission bandwidth. In the implemented WVSN, 2% of the average recognition accuracy was traded-off to minimize expenditure in terms of both sensor node overall power as well as bandwidth utilization, respectively, by 43% and 86%. As part of future work, the implemented WVSN could still be improved by training the neural network model on images with lower quality as well as using a compression algorithm that yields an even greater compression ratio. Furthermore, in the implemented WVSN, a fixed number of subimages is used, however, that can be improved by having a handshaking protocol between the sensor node and the central server node. This will allow the central server node to instruct the sensor node to stop the transmission of subimages when an acceptable recognition probability is achieved.