Ultrasound computed tomography (USCT) can visualize a target with multiple imaging contrasts, which were demonstrated individually previously. Here, to improve the imaging quality, the dynamic speed of sound (SoS) map derived from the transmission USCT will be adapted for the correction of the acoustic speed variation in the reflection USCT. The variable SoS map was firstly restored via the optimized simultaneous algebraic reconstruction technique with the time of flights selected from the transmitted ultrasonic signals. Then, the multi-stencils fast marching method was used to calculate the delay time from each element to the grids in the imaging field of view. Finally, the delay time in conventional constant-speed-assumed delay and sum (DAS) beamforming would be replaced by the practical computed delay time to achieve higher delay accuracy in the reflection USCT. The results from the numerical, phantom, and in vivo experiments show that our approach enables multi-modality imaging, accurate target localization, and precise boundary detection with the full-view fast imaging performance. The proposed method and its implementation are of great value for accurate, fast, and multi-modality USCT imaging, particularly suitable for highly acoustic heterogeneous medium.  Ultrasound computed tomography (USCT) is able to visualize an imaging region of interest with different modalities, including the quantitative spatial distribution of speed of sound (SoS), acoustic attenuation, and pulse–echo reflection images ,2,3,4,5.Although USCT provides the quantitative estimates of acoustic speed in transmission USCT and the qualitative delineation of the different acoustic impedance in reflection USCT, the inside SoS information and outside boundary structure need to be integrated to obtain a more refined target structure. This demand is prominent in the case of accurate imaging positioning with heterogeneous acoustic speed, such as in breast tumor margin identification, which should be at the sub-millimeter level to assist tumor removal surgery . Mark, AConstant speed is typically set in the delay and sum (DAS) beamforming in the reflection USCT. However, this assumption cannot match the practical acoustic heterogeneous tissue , and sucIn this work, the dynamic SoS map derived from the transmission USCT was adapted for the correction of the acoustic speed variation in the reflection USCT. The time of flights (ToFs) from the transmitter–receiver pair were selected via P-phase picker method . This hiWe briefly introduce the proposed method. The dynamic SoS-corrected transmission–reflection USCT process includes three steps: (1) The dynamic SoS map was firstly restored via an iterative method with the transmitted signals; (2) The MSFM method was used to calculate the delay time from each element to the imaging field of view on the basis of the above variable SoS map; and (3) The delay time in the conventional constant speed assumed full-view DAS spatial compounding reconstruction would be replaced by the practical measured delay time in the reflection USCT. φT) were selected via the method from ref. [js is the value of the spatially variable acoustic speed, k is the iteration step, i is the index of projection ray, j is the index of the reconstructed pixel grid (1 ≤ j ≤ N), w is the weight factor that relates the grid (j or n) to the ToFs at the projection i, λ is the relaxation factor that controls the convergence of the reconstruction (0 < λ ≤ 1), it is the i-th projection in the set T with the certain transmitted channel φ in the ring array transducer, and ic is the weighted error between actual projection data and estimated projection data, which updates at each iteration. After reshape js, the two-dimensional SoS map, which can be instantly adjusted according to the practical target that was obtained. Taking into account the situation that acoustic refraction, diffraction, and scattering co-exist in the practical target, such as tumor in the breast or bone in soft tissue, the SART algorithm was employed in the first step. It can update all rays passing through a pixel grid at a certain projection angle in each iteration; hence, the measurement errors and interference factors can be suppressed with improvement of restoration accuracy. The ToFs sjs, the MSFM method further computed the two-dimensional delay time map  between the transmitter–receiver pairs and each grid point  in the imaging field of view. It utilized the Eikonal equation to approximate the motion of the acoustic front wave, which can propagate along the orthogonal direction and diagonal direction of the grid. The details of MSFM can be seen in our previous work [On the basis of the above measured ous work . In ref.ous work , the traDASR is the beamformed US intensities at the pixel grid , and R is the raw channel data that are acquired by the transducer channel k at the sampling time t. K denotes the total number of adjacent transducer channels involved in the reflection (1 ≤ k ≤ K). A denotes the total sampling angles that are determined by the number of the ring array transducer (1 ≤ a ≤ A). Δmtn, is the time delay from each element of transducer channel k to the predefined imaging grid , which, in most cases, is computed with constant speed. mWn, is the apodization factor that reduces the cumulative intensity of beamformed US signals, which cannot be detected due to the limited reception angle of the transducer element and its corresponding acoustic field. In this study, this acoustic field can be simulated via acoustic field model and determined by practically measuring the intensities of the oppositely received signals propagated in water  calculated in the second step, the SoS adaptive DAS beamforming ismtn, is replaced by the MSFM method calculated delay map mTn,. It can be derived dynamically from the variable SoS map. With the SoS correction, calibration of the reconstructed reflection images is not necessary, which can be verified in the following simulations and experiments. In the conventional reflection mode USCT, the image is restored via DAS beamforming based on constant assumed sound speed. The weighted DAS reconstruction method with full-angle spatial compounding function is,in water . With thThe simulation models were established to validate that the multi-modality capabilities and precise imaging positioning can be realized via the dynamic SoS-corrected DAS reconstruction in the transmission–reflection USCT. The k-Wave toolbox was chosen for the transmitting and receiving of ultrasound signals, and the simulation configuration is shown in The other model had relatively complicated multi-targets to mimic the multi-layer structures in practical biological tissue, as shown in To further verify the feasibility, a relatively complex multi-target numerical model was established to study the influence of various acoustic speed differences. The acoustic speeds to recover the reflection images of pp positive pulse. The imaging sequence is the same with the simulation setting, where the first transmitting pulse was fired and then all the channels (including the transmitting channel) received the response after a certain propagation time. Here, 128 transmitting and receiving events took place to acquire the full-view US data, and the entire data acquisition time was 25.6 ms. The schematic of the ring-array-based transmission and reflection USCT experimental setup is shown in 2. In order to accurately characterize the receiving angle of the transducer element, simulation and practical tests were performed separately. The simulated transmitting acoustic field from the first element is shown in ijw and m,nW. In addition, the number of adjacent channels k around the transmit channel should also be adjusted according to the imaging quality.After data acquisition, the channel data was processed on a computer  with the proposed method in To verify the practical feasibility of the proposed method, a steel bar (diameter of 12.7 mm) with relatively large acoustic speed difference was placed in the central area of ring array ultrasound transducer. The reconstructed results shown in Preliminary research was further carried out on a human finger, which is in compliance with the research committee of Shenzhen University. The index finger of the volunteer was immersed in the water, and the imaging plane laid in between the proximal interphalangeal and the distal interphalangeal. The constant speed (1495 m/s) reconstructed reflection image is shown in Our method and its implementation can promote the ultrasound image quality in the reflection USCT with the SoS information derived from the transmission USCT. Beyond the multi-contrast imaging ability (inside quantitative acoustic speed and outside acoustic impedance difference), this approach can improve the accuracy of target positioning and boundary delineation with fast full-view imaging ability. The numerical simulations were conducted firstly to validate our assumption. Then, the results from the phantom and in vivo experiments further verified that the proposed method can be applied in the highly acoustic speed heterogeneous medium. With the custom-made ring array transducer, the large reception angle can mitigate the effect of sparse sampling with limited element number of the ring array. The restored target information can be more accurately acquired for the SoS-corrected reflection US images without additional calibration to search for the optimized constant speed. The imaging sequence with all transmitting–receiving array elements can be accelerated by the GPU, enabling full-view fast imaging capacity. Therefore, our approach and the corresponding system enable fast, accurate, and multi-modality imaging.In future work, the proposed method and system have the potential to be applied to biological tissues with large difference in acoustic speed, such as transcranial ultrasound neuromodulation and whole-body imaging of small animals. Meanwhile, the image fusion technique will be applied to co-register the SoS map, acoustic attenuation, and boundary structure. Thus, the inside acoustic speed and reflectional acoustic impedance in the boundary may be integrated. In addition, even if the reception angle is maximized in our ring array transducer, the sparse sampling effects still exist theoretically, and the compressed sensing method will be used to further conquer this effect with improvement of imaging quality. Moreover, based on the same hardware platform, the dynamic SoS map from the USCT will be incorporated in photoacoustic-computed tomography to provide additional optical contrast to reveal multiple physicochemical mechanisms.


IIC) and the correlation intensity index (CII). The epistemological component of this study is the development of methods of registration, storage, and effective use of experimental situations that are comfortable for the researcher-experimentalist in order to be able to control the physicochemical and biochemical consequences of using nanomaterials. The proposed approach differs from the traditional models based on QSPR/QSAR in the following respects: (i) not molecules but experimental situations available in a database are considered; in other words, an answer is offered to the question of how to change the plot of the experiment in order to achieve the desired values of the endpoint being studied; and (ii) the user has the ability to select a list of controlled conditions available in the database that can affect the endpoint and evaluate how significant the influence of the selected controlled experimental conditions is.Algorithms of the simulation of the anticancer activity of nanoparticles under different experimental conditions toward cell lines A549 (lung cancer), THP-1 (leukemia), MCF-7 (breast cancer), Caco2 , and hepG2 (hepatoma) have been developed using the quasi-SMILES approach. This approach is suggested as an efficient tool for the quantitative structure–property–activity relationships (QSPRs/QSARs) analysis of the above nanoparticles. The studied model is built up using the so-called vector of ideality of correlation. The components of this vector include the index of ideality of correlation ( Knowledge is the basis of all actions aimed at improving people’s lives and the evolution of civilization as a whole. However, knowledge has internal contradictions. For example, in order to manage or even observe complex processes, it is necessary first to study the available information, which is the personification of the corresponding knowledge. If the knowledge is not structured, learning to use these disordered facts or skills becomes quite expensive and difficult in several respects, such as the necessity of a long period to learn and apply expensive equipment and software. The main function of the quasi-SMILES conception examined here is to search for reasonably simple ways to study complex phenomena. The simulation of physicochemical and biochemical behavior nanomaterials is quite a complex phenomenon.Strangeness is one of the manifestations of reality about which it is difficult to speak clearly. Nevertheless, strangeness is often a property of things that are new, unexpected, or important. For the implementation of any activity, an economy of thinking is necessary. In practice, such savings can be achieved in various ways. Sorting is perhaps the simplest fragment of the thought economy process. Sorting consists of selecting the most informative of the data under experimental conditions in the laboratory, production, and interaction with environmental circumstances . However, sorting does not provide any guidance for decision making at the stage when the choice of priorities is made. Models are needed at this stage. Having a model for a process can help one to manage the process. The complexity of the choice is considerable because usefulness and harm can change places when harm becomes a benefit and the benefit turns out to be harmful. For instance, the toxicity of nanoparticles is considered a useful quality because it can be used for good aims. However, left unattended, this toxicity can harm or even kill humans and animals. The list of nanoparticles is expanding exponentially. The number of types of toxicity is by no means small. Obviously, under such circumstances, it is impossible quickly to evaluate experimentally all nanoparticles that are used or can be used. However, the assessment of the physicochemical and biochemical behavior of a significant number of new nanoparticles using databases on already studied nanoparticles is quite feasible. The key points in the development of such models are the need to reduce the memory and logic requirements of the users of the models. In other words, developers of models should provide user-friendly means of evaluating new nanoparticles. At the same time, it is highly desirable and important that such models consider the effect of possible changes in the corresponding directions for the experimental use of nanoparticles .Quantitative structure–property–activity relationships (QSPRS/QSARs) are a well-known approach to establishing models of different endpoints considered as a mathematical function of molecular structure. A successful QSAR analysis is possible if and only if: (i) there is a large enough number of compounds with a clear definition of the congeneric features corresponding to the molecules; (ii) there is a hypothesis on how and which molecular features affect the endpoint ; and (iii) checking of the predictive potential of the model can be carried out ,3. HowevThere is an urgent need to clarify the approaches and methodology for measuring the biochemical potential of engineered nanomaterials. Factually, this is a problem of tuning computational and experimental approaches oriented to “traditional” substances for application to nanomaterials. The possibility of employing computational approaches like nano-QSAR or nano-read-across to predict nanomaterial hazards based on some “standard” databases is an attractive possibility from a financial point of view. The attractiveness from an ethical point of view is also clear . Many research studies have endeavored to investigate the eco-toxicological hazards of engineered nanomaterials. However, little is known regarding nanomaterials’ actual environmental risks, combining hazard and exposure data on a planetary scale .It has been assumed that strangeness and research activity rarely intersect. However, when they meet, they either reinforce or disregard each other. For example, modelling, one of the most important and complex areas of research, can be summed up in the short aphorism “All models are wrong, but some are useful” .Systematization of knowledge related to nanomaterials has become necessary due to the fast growth of applications of these “unusual” substances. Systematization involves various aspects of research activity. The development of approaches that allow for the simulation of different characteristics of nanomaterials, including their interactions with other species, is one of them. There are many methods to perform such simulations. One of the possible approaches is to carry simulations out using the so-called quasi-SMILES ,11,12,13The dataset used in this study includes measurements of half maximal effective (EC50), inhibitory (IC50), and lethal (LC50) concentration toxicity endpoints toward cell lines A549 (lung cancer), THP-1 (leukemia), MCF-7 (breast cancer), Caco2 , and hepG2 (hepatoma) under different experimental conditions  for human cells. The indicated conditions and circumstances were represented by special codes listed in The listed codes for quasi-SMILES make it possible to constructively describe the available experimental situations for developing models in order to predict the results of varying codes . The system described can assess the statistical significance of individual experimental conditions . In other words, concentration values, exposure times, impacted objects, nanoparticle sizes, and others are under consideration to simulate the behavior of nanoparticles.After removing duplicates, the source  containshttp://www.insilico.eu/coral, accessed on 29 May 2023). The values of the optimal descriptor serve as the basis for the model of half-maximal concentration (HMC)  calculated by the formula:The optimal descriptor is the sum of the correlation weights of the quasi-SMILES codes obtained by the Monte Carlo method using the CORAL software . N is the number of epochs of the Monte Carlo optimization.The optimal descriptor depends on the selected method of the Monte Carlo optimization of the correlation weights for codes of quasi-SMILES . The T aThe optimal descriptors are calculated using the correlation weights obtained by the Monte Carlo optimization ,17. Two ATr and PTr are correlation coefficients between the experimental and predicted values for the active and passive training sets, respectively. The IIC represents the index of ideality of correlation .[O-2].[O-2].[Al+3].[Al+3]), thereby increasing the number of optimized parameters. Of course, such changes do not guarantee an improvement in the predictive potential of the model, but they do provide the user with extended opportunities in the search for a model of the phenomenon and perhaps even stimulate the user’s creative activity. Another important although hidden point is that the considered approach allows the user to identify and discard those quasi-SMILES fragments that are non-informative due to their low prevalence in training samples and/or in the general array of available data. This defines automatically through the appropriate selection of the threshold described above ). Since QSAR is actually a random event [On the other hand, the user can expand the brutto formulas by representing the corresponding metal oxides with traditional SMILES  [IIC is a correlation with forced minimization of the mean absolute error (MAE). It is to be noted, however, that the application of IIC gives an improvement to the statistical characteristics for calibration and validation sets which is accompanied by reducing the correlation coefficient for the training sets. This is a paradox situation. Nevertheless, from a practical point of view, this situation is preferable to overtraining . An analysis of the graphical representations of such a paradox observed with various geometric configurations on plots for ‘experiment vs calculation’ shows that such idealization is not always possible; fortunately, however, it is possible in the majority of cases of the different arrangement of points on the plot diagram ‘experiment vs calculation’ [A very significant component of models built on optimal descriptors using quasi-SMILES codes is optimization procedures by the Monte Carlo method. The choice of the target function is the key to the success of such Monte Carlo calculations. The ideality index of correlations (IIC)  turned oIIC) . Ideal culation’ .CII) [CII.Another useful invention for improving the predictive potential of models based on quasi-SMILES codes is the Correlation Intensity Index (CII) . Data onIIC and CII, improves the predictive potential of the studied models but in detriment to the statistical quality of the models on the training set. The described approach can be easily adapted to simulate other experimental situations and endpoints for nanomaterials and other substances .The present study demonstrated that the quasi-SMILES technique gives statistically robust models for the half-maximal concentrations for the five cell lines. We showed that the statistical quality is well reproduced for five random splits of available data into a structured training set  and an external validation set. Such approach is tested and recommended for various applications of the quasi-SMILES approach. Paradoxically, the vector of ideality of correlation, which is the sum of the described IIC and CII cannot only improve Monte Carlo optimization, but the mentioned values can also be indicators of the predictive potential of various models.A quasi-SMILES approach describing experimental situations can be modified both by feedback  and purely heuristically in accordance with spontaneous ideas for which statistical expertise is possible. Thus, quasi-SMILES are a simple and versatile approach for modelling experimental situations not yet implemented in practice. Indices


Disassembly of electric vehicle batteries is a critical stage in recovery, recycling and re-use of high-value battery materials, but is complicated by limited standardisation, design complexity, compounded by uncertainty and safety issues from varying end-of-life condition. Telerobotics presents an avenue for semi-autonomous robotic disassembly that addresses these challenges. However, it is suggested that quality and realism of the user’s haptic interactions with the environment is important for precise, contact-rich and safety-critical tasks. To investigate this proposition, we demonstrate the disassembly of a Nissan Leaf 2011 module stack as a basis for a comparative study between a traditional asymmetric haptic-“cobot” master-slave framework and identical master and slave cobots based on task completion time and success rate metrics. We demonstrate across a range of disassembly tasks a time reduction of 22%–57% is achieved using identical cobots, yet this improvement arises chiefly from an expanded workspace and 1:1 positional mapping, and suffers a 10%–30% reduction in first attempt success rate. For unbolting and grasping, the realism of force feedback was comparatively less important than directional information encoded in the interaction, however, 1:1 force mapping strengthened environmental tactile cues for vacuum pick-and-place and contact cutting tasks. As a result of the increasing demand for electric vehicles (EVs) , a largeTelerobotics aims to mitigate the hazards of disassembly by enabling a human operator to carry out disassembly tasks remotely through a local interface while imparting some of the human operator’s dexterity and fine motor control to the robot. Haptic devices are commonly employed in telemanipulation studies due to their ability to deliver force feedback to the operator while carrying out tasks. Nevertheless, one key challenge in haptic devices can be the master-slave asymmetry in regards of the kinematics  and a relatively low-cost platform using a haptic device paired with a single cobot. We examine causative factors for differences in performance between these two platforms for each task.The remainder of the paper is structured accordingly: Methods of battery disassembly can be broadly categorised into fully-manual, fully-autonomous, and semi-autonomous approaches. Given the hazards that EV battery disassembly environments pose against human operators, the literature has been more oriented towards developing autonomous and semi-autonomous approaches on the basis of potential improvements in efficiency and safety. Autonomous approaches aim to increase the efficiency of disassembly by allowing robots to plan and carry out repetitive tasks in unstructured environments through the use of visual and tactile feedback. Examples of such works can be found in Recently, human-robot collaboration (HRC) has garnered attention for disassembly of end-of-life products, integrating both the robot’s high efficiency in repetitive tasks and the human flexibility with higher cognition. However, there are few studies on this subject in the literature. In , a high-Telemanipulation has been commonly employed in the nuclear industry for delicate handling of wastes that present a radiation hazard for human personnel. For example, the use of gloveboxes is examined in The majority of considered studies have focused on telerobotics for assembly tasks, nuclear decommissioning and waste handling applications. However, there is a paucity of research surrounding the use of telerobotics in an EV disassembly context. The authors in This study focuses on the Nissan Leaf 2011 battery pack as a case study to demonstrate a range of disassembly tasks using telemanipulation. The Nissan Leaf 2011 pack comprises 192 cells enclosed within 48 modules, arranged with two forward vertical stacks of 12 modules and a single rear horizontal stack of 24 modules. An overview of the main disassembly stages for pack-to-cell disassembly is given in Based on the disassembly sequence in A common framework to accomplish telemanipulation tasks is to use a lower-cost, handheld platform such as a haptic device in combination with a robot in a master-slave configuration. In this case, we consider the case of a Phantom Omni haptic device and a Panda cobot . Due to Alternatively, in this work we consider a platform consisting of two identical Franka Emika Panda cobots operating in a master-slave configuration. With this approach, the user directly manipulates the master Franka arm, whose motions are directly mirrored to that of the slave arm. Owing to the identical configuration of both robots, both joint position and torques can be mapped between the robots 1:1. This 1:1 mapping results in natural and responsive feedback being delivered to the user. However, this also poses hazards as the user will be potentially exposed to the full forces involved in a specific task. The joint space control of both arms furthermore has advantages in the problem of singularities and joint limits, which are present in mapping the Cartesian end-effector pose of the master to the task space pose of the slave robot. However, the lack of task-space control has potential drawbacks when accomplishing tasks along specific task directions, such as removing bolts or cutting, more difficult and less intuitive for the user.N-link manipulator in joint spaceq and command torques τ by subscripts l, f for master and slave respectively.To achieve a virtual coupling between master and slave devices, we first consider the dynamic equation of a rigid ql, qf cannot be achieved in practice due to a mismatch in the number of kinematic degrees of freedom and joint ranges compared with the Franka arm. Hence, it is required to either map the joint space onto a reduced subset of the full joint space of the robot, or operate in a mutual task space. In this work, 6DoF Cartesian mapping is employed for motion control of the slave Franka arm with the Phantom Omni master arm. As the human operator moves the master arm in its workspace, delta Cartesian pose PTΔ is computed and then mapped to the end-effector delta pose FTΔ via the workspace transformation FTΔ, and PTΔ represent the delta transformation matrices for Franka and Phantom respectively. Position and orientation components of PTΔ are assigned to the desired velocity ex = xf − xt for the slave arm. Thus, the slave arm control law is:Kp, Kd are controller stiffness and damping matrices respectively. This results in the desired closed-loop dynamic behaviour from Eq. Fext experienced at the Franka end-effector is transformed with respect to the Phantom so the user is able to perceive it as force Fl as follows:G = 0.1. This factor was determined experimentally by comparing the maximimum expected force across all tasks from preliminary data and scaling to the maximum force capabilities of the haptic device (3.3N). To distort the force feedback to a minimal extent, and to maintain consistency with the constant 1:1 feedback of the identical cobot setup, this factor was held constant across all trials. A necessity of the task space control scheme is the remediation of singularities and joint limits. In this case, a null space position regulation term is added to (3) to avoid joint limits. Unlike pseudoinverse control where large solution values can be obtained, with the controller design in Eq. τf tending to zero along the singular directions.Force feedback is an essential part of a bilateral teleoperation system in which a force feedback is maintained at control frequency. This helps the user to have a tactile perception of the slave robot’s environment. In a similar way to (2), the external force vector c device .3N. To deq = qf − ql is the joint space error. This results in the closed-loop dynamics in joint space for the slave armC, g are subject to uncertainty, which results in reduced tracking performance of both controllers and the presence of steady-state error. Tuning of the control gains Kp, Kd was accomplished similarly to G via preliminary experiments, similarly to other comparative studies such as Kp was increased for each of the control strategies on the slave arm in isolation before instability occurred, while Kd was set to obtain a critically damped behaviour. This provides the best tracking performance in free space and reduces the influence of uncertainties in the model parameters C and g. For the joint control strategy, the gains were scaled down according to the torque capabilities of each joint. Due to the combination of higher forces applied to the master with the low stiffness of the human operator, an additional damping term Kd,l is added to the response of the master. The value of Kd,l was selected by identifying the minimal damping value assigned to all joints to suppress oscillations arising from feedback effects from the bilateral master-slave coupling.For the Franka arm, a joint impedance control scheme is used that directly maps the joint configuration of the master arm to the slave arm . The conA contiguous sequence of tasks relating to the disassembly and extraction of a stack of modules from the Nissan Leaf 2011 battery pack were identified as feasible targets for robotic disassembly, presented in For each task, the operator is provided with two fixed camera views of the module stack, and otherwise does not have a direct view of the scene. These views are held constant for all tasks with the exception of cutting, where the operator is instead provided two camera views of a material holder containing the cutting workpiece. Prior to commencing each task, both master and slave robots are initialised to a home position in joint space held constant for all tasks. As the focus of this work is not on human factor analysis and subjective measures of performance based on end-user evaluations, but on objective measures of task performance, the operator is assumed to have familiarity and prior experience with both teleoperation platforms. In addition, each operator was given 10 min of training with each task with both telemanipulation systems before first attempting each task. Four expert operators participated in the study. For each trial, a single operator was selected randomly to perform the task. A more detailed description of each task is provided as follows.This task considers the unbolting of a set of 4 fasteners constraining an individual stack of modules. The operator has access to a motorised universal socket wrench tool mounted at the robot’s wrist. The operator is responsible for aligning the socket wrench with the bolt and operating the tool to unscrew the fastener. The task is considered successful if the bolt is removable by hand without any further unscrewing action. If the success condition is not met after the first attempt, or the configured force thresholds (40N) of the robot are exceeded, the task is considered a failure.This case study follows from the previous unbolting stage by removing each fastener from the stack before removal of the cover. For this task, the Franka hand was used with a configured grasping force of 50N. The operator was required to remove a set of 8 fasteners by maneuvering the hand towards each of the fasteners, grasping and removing the bolt from the stack and depositing into a container. Due to the limitation of the camera views of the object, the operator must use a mixture of tactile and visual exploration to successfully remove the fasteners. For each bolt, the failure condition is met if the bolt is not grasped after the first grasp attempt, or if the grasp is lost outside of the target container.After the removal of the fasteners, the operator must then remove the module cover plate to access the underlying module stack. Due to the weight and geometry of the cover plate, it is essential for the operator to find and manoeuvre towards a good grasp point to ensure safe transportation of the cover during removal. The configured grasping force was increased to 60N for all experiments. The failure condition is met if the first grasp is unsuccessful or the grasp is lost during the transportation of the cover.This case study considers unstacking and sorting of the EV battery modules using a vacuum suction gripper. The operator must remove a pair of modules from a stack and deposit them into a container. This task requires visual positioning of the gripper onto a suitable surface for grasping, while furthermore maintaining contact to engage the suction cups with the material without exceeding the force limits of the robot. Similarly to the cover removal case study, the failure condition is met if the first grasp is unsuccessful, or if the grasp is lost during transportation.Additional semi-destructive disassembly tasks may be carried out to further disassemble the individual modules to access the Li-ion cells or to separate and remove fasteners or connectors that are not amenable to non-destructive disassembly. We consider the case study of contact cutting of a planar material along a predefined visually marked desired path using a slitting saw tool. Due to the limited availability of battery materials and the power limitations of the spindle, we consider cutting a cardboard sheet as a benchmark for the more general process of cutting different planar materials. The operator must use tactile feedback to control the cutting force while also using visual feedback to achieve precise positioning along the path. The desired cutting path consists of a window of ±2.5 mm measured from the centroid of the marked path. If the cut deviates from the path greater than this threshold, or if the cut is incomplete along parts of the desired path, the failure condition is met.success rate and average task completion time over all trials are summarised in For each case study, the overall In From an overview of unbolting of the fasteners using the wrench tool. Similarly, bolt removal task in which fasteners are grasped, lifted up from their holes, and placed in a container; thus, the position z spikes in the red region (“place” stage). Interestingly, despite the differing scale of force feedback between the two platforms, the magnitude of external force observed over time for the bolt removal task is similar for both platforms. This suggests that the realism and scaling of the force feedback is less important for accomplishing precise grasping tasks for disassembly than the directional guidance provided by the force feedback during contact.Some tasks involved placing components after disassembly into a waste/recycle bin, like in the cover removal task, where most of the time was spent on post-grasp manipulation, i.e., the “place” stage. Operators spent on average 39, and 37 s placing the objects which averaged 56%, and 35% of the task completion time with cobot and haptic platform, respectively. Around 30% of the time was spent in the “place” stage for the cobot platform; due to the lack of Cartesian control, the operator reached joint configurations where large reconfigurations were required between tasks to access the workspace. This is evident in Z direction). Furthermore, the cover exhibits a more dispersed mass distribution in contrast with the module, where a majority of the weight is concentrated about the grasp point. Hence, positioning of the cover imposes a greater demand due to higher torque required to rotate the joints, which is largely abstracted for the Cartesian controlled haptic-Franka platform. This is corroborated by the more gradual and tortuous path seen in For the module sorting task, significant improvements in the task completion time were observed, with an SMD of 2.31. Similarly to other tasks, improvements in coarse and fine alignment speed are observed, although the reduction in completion time is driven by time reductions over all stages of the task, including the “action” stage. Examining in positional mapping afforded with the Franka, rather than improvements in realism of the tactile feedback. However, during the cutting task, the realism of feedback is comparatively more important. This can be examined in For cutting, a small SMD of 0.41 suggests the effect on task completion time for the Franka platform is marginal relative to the trial-to-trial variation. Cutting is a complex task dependent on a wide range of material and tool-specific parameters, and a wide variation in completion times is observed between task trials. For the Franka, a greater proportion of time is spent performing the cutting task proper. This is on average achieved faster with the haptic device, however, a much greater proportion of time (55% vs. 32%) is required for initial alignment of the tool with the desired path. The fine alignment stage in particular is unique for the cutting task in that the operator must rely purely on visual alignment with the desired path; this further suggests the reduction in completion time for the presented tasks is in large part related to the expanded workspace and 1:1 The increased feedback capability of a setup with identical cobots has some notable disadvantages. From For the module stack disassembly case study comprising 4 modules, total disassembly times averaged 25.8 min and 14.7 min for the haptic and Franka platforms respectively, representing a 43% reduction in overall completion time when using two identical cobots. Considering module cover separation consisting of 4 linear cutting operations, it is estimated to require 8 min per module to expose the battery cells, reducing by 25% for the identical cobot case. Overall disassembly times demonstrate that while disassembly via telemanipulation is feasible, several efficiency improvements and assistive strategies are required to improve disassembly speed. However, there is large variation in disassembly time estimates across battery models and techno-economic analyses in literature. For example, an estimate of 12 s for manual unscrewing and 6 s for manual removal of free components across multiple battery designs  implies Limitations of the present work are that only objective measures of task performance are considered under the assumption of familiarity with the necessary tasks and teleoperation platforms (expert operators). However, to obtain a holistic comparison it is necessary to consider operators of different experience levels, as well as consideration of subjective performance measures, such those established in NASA TLX that establish measures of cognitive load and effort for the sequence of disassembly tasks. Furthermore, while the semantic breakdown of the demonstrated tasks can provide insights into the most challenging parts of each task, the effect of variable autonomy on task performance with respect to each platform could still be explored. For example, differences in success rate for tasks such as bolt removal could be addressed by guiding the user towards suitable grasp points. Therefore, future works will consider the effect of shared control strategies on the completion time and success rate of the presented disassembly tasks.This study demonstrates the telerobotic disassembly of a stack of modules from the Nissan Leaf 2011 battery pack. A comparative study between a master-slave setup consisting of a haptic device paired with a cobot, and two identical cobots examined the success rate and completion time for accomplishing unbolting, extraction of bolts, grasping and removal of the cover plate, sorting of modules with a suction gripper and contact cutting. While overall success rate was higher with the haptic device, a setup with identical paired cobots was found to significantly improve task completion times and time consistency across the entire set of disassembly tasks. This suggests that quality and realism of force feedback is comparatively less important for accomplishing precise manipulation tasks, but instead the 1:1 position mapping between master interface and slave and expanded workspace were main contributors to the efficiency improvements. While 1:1 mapping of torques between identical cobots was beneficial for grasping with a vacuum gripper and cutting, chiefly due to the enhanced realism of the interaction, providing strengthened tactile cues to the operator, this further had detrimental effects on operator physical effort and quality of cut. While a semantic breakdown of the presented disassembly tasks and objective performance measures provides insights into the key stages, feasibility and difficulty of each disassembly task with each platform, the effect of differing levels of operator expertise, subjective task load measures and variable autonomy on task performance was not considered. Further work will focus on evaluating the effect of variable autonomy on task completion performance in the context of the presented disassembly case studies.


The minimum-energy formation strategy for interconnected networks with distributed formation protocols is persented, where the impacts of the total energy restriction and the interaction silence are analyzed, respectively. The critical feature of this article is that the distributed formation and the minimum-energy restriction are realized simultaneously, and the total energy restriction is minimum in the sense of the linear matrix inequality. However, the guaranteed-cost formation strategy and the limited-budget formation strategy cannot guarantee that the energy restriction is minimum. Firstly, sufficient conditions for minimum-energy-restriction formation without the interaction silence are proposed, which can be solved by a specific optimization approach in terms of the linear matrix inequality, and the formation whole motion trajectory is determined, which is closely related to the average of the initial states of all agents and formation control vectors. Then, minimum-energy-restriction formation criteria for interconnected systems with the interaction silence are proposed by introducing two inhibition parameters and the interaction silence rate. Finally, two simulation examples are performed to illustrate the effectiveness of theoretical analyses. Especially, the investigation of the formation behavior of interconnected systems inspires by the distributed mechanism research of the flocking and consensus behaviors, where the flocking and consensus behaviors mainly focus on the theoretical analysis, but the formation behavior has extensive applications in practical interconnected systems, such as the coordinated attack of multiple drones and the cooperative operation of multiple intelligent robots.The cooperative behaviors of interconnected systems attracted much interests in various research fields, such as the flocking behavior17. Furthermore, the energy restriction is a necessary component for practical applications of interconnected systems and can be categorized into two types: a local energy restriction and a total energy restriction. Semsar-Kazerooni and Khorasani18 proposed a local restriction function on the basis of the partial information interaction and realized an optimal cooperation. Vamvoudakis et al.19 presented a multi-agent differential graphical game approach and gave an online adaptive learning solution with a local restriction function. A total restriction function associated with the linear quadratic regulator was proposed in20, where the optimal cooperation can be achieved when the information topology is complete. In23, the guaranteed-cost strategy was proposed, where the different upper bounds of the total energy restriction were obtained, but the error between these bounds and the minimum bound cannot be determined. The limited-budget strategy was presented in24, where an upper bound of the energy restriction was specified in advance. However, these two upper bounds of the total energy restriction may not be the minimal value, and both the guaranteed-cost and limited-budget formation strategies cannot guarantee that the total energy restriction is minimum.It is necessary for distributed formation control that all agents maintain a specific structure to finish some tasks via a partial information interaction, as shown inet al.25 proposed an observer-based formation strategy that takes into account the interaction silence. For heterogeneous linear interconnected systems, sufficient conditions for output formation-containment were presented in26, where an approach was proposed to optimize the interaction silence rate. Yang et al.27 explored the adaptive control strategy for interconnected systems with the interaction silence and theoretical results were applied to cooperative teleoperators. Wang et al.28 established a new interaction silence framework for heterogeneous coupling networks using the linear matrix inequality technology. With the interaction silence, it is meaningful and challenging to minimize the total energy restriction of interconnected networks by regulating the control gain matrix of the distributed formation protocol. To the best of our knowledge, distributed formation control problems for interconnected systems with the minimum energy restriction and the interaction silence are not comprehensively investigated and are still open.Moreover, in the above literatures on distributed formation control with the energy restriction, it was required that the information topology is connected or has a spanning tree and must maintain the continuous transmission of cooperative information. However, there may be instances where the information transmission fails due to the channel jamming or the anti-jamming. Information interactions of interconnected systems may disappear and recover when there is the electronic interference in the communication environment. Furthermore, the active silence strategies may be adopted by the interconnected system to save the energy or to avoid the detection of some electromagnetic spectrum equipment. In this case, there exists no information interaction among agents in some time segments. Qin This article focuses on minimum-energy-restriction formation achievement problems with the interaction silence in interconnected systems. A distributed formation protocol is shown to realize minimum-energy-restriction formation control, where a total energy restriction term is imposed. Furthermore, sufficient conditions for minimum-energy-restriction formation without the interaction silence are proposed by establishing the relationship between the energy restriction and the gain matrix. These conditions are scalable since the dimension of the matrix variable is not related to the number of agents and can be verified by the generalized eigenvalue method. Especially, the formation whole motion trajectory is obtained, where the effects of the initial states and formation control vectors are depicted, respectively. Moreover, interconnected systems with the interaction silence are dealt with, where two inhibition parameters and the interaction silence rate are introduced to restrict the divergent quantity during the interaction silence subsegment, and the associated criteria are proposed to realize formation control with the minimum energy restriction. It should be pointed out that the interaction silence is essentially the same as the intermittent control from the point of the information interaction in interconnected networks. However, from the point of the control method, some event-triggered control strategies can also lead to the intermittent control.24 cannot minimize the total energy restriction of all agents as a whole. The second aspect is that the effects of the interaction silence on the formation control with the minimum-energy restriction are handled by introducing two inhibition parameters and the interaction silence rate in this article, and a specific condition about the interaction silence rate is constructed to constrain the divergent quantity during the interaction silence subsegment. However, the influence of the interaction silence on the minimum energy restriction was not considered in24. The third aspect is that the explicit expression of the whole motion trajectory for the minimum-energy-restriction formation is determined, where the effects of the initial states of all agents and formation control vectors are depicted and it is shown that the interaction silence does not impact the formation whole motion trajectory, which were not settled in24.Compared with the existing literature of optimization formation, the contributions and innovations of this article own the following three novel aspects. The first aspect is that this article realizes the distributed formation control and the minimum-energy restriction simultaneously. However, the guaranteed-cost strategy and the limited-budget strategy inThe rest sections are to be outlined as follows. Without the interaction silence, the problem statement for formation control of interconnected systems with the minimum energy restriction is presented in “The key notations used in this work are shown in the following Table M homogenous agents as follows:M vectors Consider an interconnected system with G, where the vertex in the graph stands for an individual agent, its edge represents an interconnected channel and its edge weight denotes the channel strength between two neighboring agents. The information topology of interconnected systems is undirected and connected, which means that two neighboring agents can acquire the cooperative information with each other simultaneously and there does not exist an agent unconnected with other agents. Especially, there is not a self-loop in this information topology G, which means that each agent can acquire its cooperative information directly without any communication actions. If agent ikk is called a neighbor of agent i. The set containing all neighbors of agent i is called its neighbor set, which is denoted by L, whose row sum is zero as shown in29; that is, The information topology of this interconnected system is described by a graph The information topology in this article is undirected and connected.i and k, E is the overall energy restriction. This formation protocol is distributed since each agent regulates its cooperative state by synthesizing the cooperative information of its neighboring agents and all agents have the same cooperative control gain. The following is the definition of the minimum-energy-restriction formation.The formation protocol of each agent is usually composed of the cooperative states and the formation control vectors of neighboring agents of each agent. We construct a formation protocol with the energy restriction as follows:minimum-energy-restriction formation achievable by formation protocol (P1) if there exists a gain matrix K such that For any bounded disagreement initial conditions mentclass2pt{minimThe objective of this article is to design the cooperative control gain such that interconnected system  achieves23 and24, respectively. For the formation control with the guaranteed-cost strategy, it is needed to give the upper bound of the guaranteed cost and the conservatism of this bound cannot be determined since it is difficult to calculate the error between this bound and the minimum bound. Although an upper bound is previously given for the limited-budget formation control, it cannot be guaranteed that this value of the upper bound is minimum. It is interesting to minimize the total energy restriction and to decrease the energy assumption of all agents as a whole. To make the energy restriction be minimum when formation is achieved, we need to deal with two key challenges. The first challenge in the formation process is to construct the relationship of the energy restriction and the gain matrix, which means that the energy restriction can be regulated by designing the different gain matrices. The second challenge is to develop a specific optimization approach such that the total energy restriction is minimum. This article focuses on minimizing the total energy restriction using the linear matrix inequality strategy.There are two main types of control strategies that are commonly used for controlling the formation of the interconnected system with the total energy restriction; that is, the guaranteed-cost strategy and the limited-budget strategy, which were addressed inIn this section, the relationship between the energy restriction and the gain matrix is constructed by the matrix variable of the Lyapunov function. Then, sufficient conditions for minimum-energy-restriction formation design and analysis are proposed, respectively, which can be obtained by a specific optimization approach.L are real and there exists an orthogonal matrix U such thatU is M-dimensional column vector with the first component 1 and 0 elsewhere. Since U is orthogonal, one hasLet mentclass2pt{minimAccording to the above state decomposition and the linear matrix inequality technology, a sufficient condition for the formation achievement with the minimum energy restriction is presented by introducing the formation feasible condition. In the linear matrix inequality sense, the minimum energy restriction can be guaranteed and can be solved through the generalized eigenvalue optimization approach.Interconnected system  is minimU is orthogonal. Due toLetFurthermore, we give a design approach of the gain matrix such that the formation of interconnected system  with formentclass2pt{minimU is orthogonal and 30, if In addition,, the impacts of the energy restriction on formation achievement are dealt with. By formation protocol P1), one has, one has31 and32, respectively, and they have the lower computation complexity compared with solving their precise values.In Theorem K, the following conclusion can be derived directly, which presents a sufficient condition of minimum-energy-restriction formation analysis for interconnected system /M. By , 9) and and\docu0)/M. By , one canmentclass2pt{minimIf interconnected system  with for30. It should be noted that the positive scalar is minimum in the sense of the linear matrix inequality, but it is not a globally optimal solution. Furthermore, the formation feasible condition 33. Especially, the formation feasible condition shows that the physical structure of each agent is an important factor for formation feasibility. Moreover, Corollary 2 determines the formation motion trajectory of an interconnected system as a whole, where the effects of the initial states and formation control vectors are given, respectively. It can be found that the formation whole motion trajectory is related to the average of initial states, which is essentially determined by the fact that the information topology is undirected and connected.In order to establish the relationship between the control gain matrix and the energy restriction in Theorem In this section, a formation protocol with the interaction silence is proposed, where interconnected channels among all agents are removed during the interaction silence subsegment. Then, by introducing the interaction silence rate and two inhibition parameters, the divergent property during the interaction silence subsegment is handled and sufficient conditions for minimum-energy-restriction formation are discussed.There may exist the channel jamming or active anti-jamming, so the non-periodic interaction silence may appear in interconnected channels among agents. In other words, all information transmissions are removed in a passive or active manner. The whole time interval can be divided into infinite and non-overlapping time segments A is Lyapunov unstable. Intuitionally speaking, the formation achievement of interconnected system : Consider six agents communicating with each other by the information topology showing in Fig. i can receive the state information of agent k directly, then the channel strength : The information topology of this interconnected system with the interaction silence is also depicted in Fig. (The initial states and formation control vectors are set as:The silence subsegment and the non-silence subsegment are set as K by We consider the distributed formation control with and without the minimum energy restriction where With the non-periodic interaction silence and the minimum-energy-restriction, the states of six agents within mentclass2pt{minimWith the total energy restriction and the interaction silence, a distributed formation protocol for interconnected systems was proposed to realize formation control under the condition that the total energy restriction is minimum. By constructing a common Lyapunov functional candidate, sufficient conditions for minimum-energy-restriction formation of interconnected systems without the interaction silence were proposed. This method ensures that the energy restriction is minimum and can be solved efficiently using the generalized eigenvalue methods. Moreover, an explicit expression of the formation whole motion trajectory was given, which contains two independent parts: the trajectory associated with the average of the initial states of all agents and the trajectory associated with the average of formation control vectors of all agents. Furthermore, the impacts of the interaction silence on minimum-energy-restriction formation were determined by introducing two inhibition parameters and the interaction silence rate, where it is essentially required that the convergent quantity during the information non-silence subsegment is larger than the divergent quantity during the interaction silence subsegment.


Grain size has a significant effect on the mechanical properties of metals. It is very important to accurately rate the grain size number of steels. This paper presents a model for automatic detection and quantitative analysis of the grain size of ferrite–pearlite two-phase microstructure to segment ferrite grain boundaries. In view of the challenging problem of hidden grain boundaries in pearlite microstructure, the number of hidden grain boundaries is inferred by detecting them with the confidence of average grain size. The grain size number is then rated using the three-circle intercept procedure. The results show that grain boundaries can be accurately segmented by using this procedure. According to the rating results of grain size number of four types of ferrite–pearlite two-phase microstructure samples, the accuracy of this procedure is greater than 90%. The grain size rating results deviate from those calculated by experts using the manual intercept procedure by less than Grade 0.5—the allowable detection error specified in the standard. In addition, the detection time is shortened from 30 min of the manual intercept procedure to 2 s. The procedure presented in this paper allows automatic rating of grain size number of ferrite–pearlite microstructure, thereby effectively improving the detection efficiency and reducing the labor intensity. G is the grain size number; M is the magnification used; P is the grain boundary intersection count; L is the intercept length.Grain size is a parameter used to describe the dimensions of grains of polycrystalline materials. It has a significant effect on the properties of metals . When porocedure . In the At present, the grain size number in the intercept procedure is mainly rated by manual measurement and calculation. Manual rating has obvious disadvantages. First, it has great subjective influence. Only experienced professionals can accurately identify grain boundaries and judge intersections, and human subjectivity during the measurement affects the rating accuracy. Second, it has low efficiency. For a metallographic image, if the intercept procedure is used, it will take at least 30 min for measurement of hundreds of intersections in multiple fields of view and for the subsequent calculation. Third, it affects health. The rating of grain size number is analyzed under a microscope, and long-term work under high-intensity light will affect the health of testing personnel.In recent years, the continuous development of artificial intelligence technology has achieved remarkable results in various fields. Deep learning, as the leading artificial intelligence technology, has attracted increasing attention and has achieved excellent performance in medical image classification and text recognition ,8,9,10. Currently, scholars have studied the research of using image-processing techniques to identify grain boundaries and calculate grain size ,14, suchThere are four facts relevant to the grain size testing of ferrite–pearlite in steel mills. (1) Almost all the steel mills use optical microscopes when testing the grain size. (2) Under the optical microscope, the grain boundary in the ferrite has obvious contrast difference from the grain inside, so such grain boundaries can be easily identified with human eyes. However, for pearlite, due to low resolution, the boundaries of pearlite colonies and the slices of ferrite–cementite cannot be seen. Hence, the pearlite grain size (the size of pearlite colonies) cannot be tested under the optical microscope. (3) Therefore, when the steel mills test the actual grain size of ferrite–pearlitic steel, almost all of them test only the grain size of the ferrite, while the pearlite part is ignored, and finally the grain size of the ferrite part is taken as the overall grain size of the sample. (4) The difficulty of testing the grain size using this method (ferrite only) is different for steels with different carbon contents. For steel with less carbon content, the grain size of ferrite can be easily tested directly because ferrite dominates in the sample and the whole sample indicates a great ferrite region with very little pearlite. However, for steel with high carbon content, since pearlite is randomly and approximately uniformly distributed throughout the sample and there is no large pure ferrite area, it is difficult to directly test the grain size of ferrite. In such cases, the presence of pearlite will significantly interfere with the efficiency and accuracy of ferrite grain size testing, and the efficiency of manual calculation of the grain size is extremely low.Based on the four facts above, when evaluating the grain size of ferrite–pearlite steels by testing the ferrite grain size, in order to solve the interference of pearlite regions to the testing process and results of ferrite, and to improve the testing efficiency, we propose a new and simple method based on artificial intelligence techniques. Since the steel mills ignore the pearlite and consider the grain size of ferrite as the overall grain size of the sample. Then, we assume that all the pearlite areas are ferrite. At this point, the exact size of grains in the pearlite (considered as ferrite) region is unknown, but it can be predicted according to the confidence of the average grain size of ferrite  using the model . In this way, we, (1) identify and test the grain boundaries in ferrite, (2) treat pearlite as ferrite, and use our model to predict the “hidden grain boundaries” inside it, and (3) count and calculate the intersections in the actual grain boundaries in ferrite and the predicted “hidden grain boundaries” in pearlite using the three-circle intercept procedure. Then, statistical calculation is made to achieve automatic rating of grain size number.Image segmentation is the first step and also the most important and difficult part of image analysis. Image segmentation is the process of partitioning an image into several disjoint parts by grayscale, color, spatial texture or other characteristics , so thatU-Net is a variant of fully convolutional neural network (FCN) . Its arcIn the contracting path of U-Net, each contraction consists of a 2 × 2 max pooling layer (with stride 2) and two 3 × 3 unpadded convolutional layers. Each unpadded convolutional layer is followed by a ReLU activation parameter for downsampling of original data. At each downsampling step, the number of feature channels is doubled. In the deconvolution of the expanding path, every step consists of a 2 × 2 unpadded convolutional layer  and two sets of 3 × 3 unpadded convolutional layers. At the same time, each deconvolution is followed by an addition of the feature map cropped from the corresponding contracting path (clipped to maintain the same shape). The final layer of the network is a 1 × 1 unpadded convolutional layer that is used to map the 64-channel feature vector to the desired number of classes. Finally, the U-Net used in this paper has 23 layers in total. The advantage of U-Net is that it has no requirement on the shape and size of the input image, especially when extremely large images are processed. In this paper, cross entropy is used as the loss function of image segmentation, and its principle is as follows:Assuming the number of samples in the training dataset is n, the cross-entropy loss function is defined as:The two feature vectors output by U-Net are the category label vector A total of 200 metallographic samples with a diameter of 10 mm were prepared with hot-rolled ribbed bars as the test samples. After grinding, polishing and corrosion, the samples showed clear ferrite–pearlite two-phase microstructures and ferrite grain boundaries.The LabelMe platform was used to distribute grain size images for experts, and the circle intercept procedure was used for the measurement and statistics of intersections and the calculation of grain size. It should be noted that hidden boundaries do not need to be marked manually, except for visible contour lines. For deep learning, the size of the dataset has a great impact on the effect of the model . A too-s be the coordinates of the upper left corner of the rotated image,Random rotation and scaling mean randomly rotating the training image by random angle or scaling by random magnification for data augmentation. For a training sample, the augmented sample obtained by image flipping and image rotation can enable the model to learn features that are not deformed by rotation during training, while the augmented sample obtained by image scaling can better enable the model to realize multi-scale training. Let The results are shown in Random erasing is a new data measurement technique, which erases the pixels with random values in the randomly selected rectangular areas in training, so as to obtain more occlusions or erase pixel values and images with different areas for data enhancement. This technique can reduce the risk of fitting and make the model robust to occlusion and erasure. The results are shown in The algorithm used in this paper falls into five modules: metallographic image acquisition module, contour extraction network module, visible intersection statistics module, hidden intersection statistics module and automatic rating module.The image acquisition module measures the metallographic samples and calculates their grain size number by the conventional three-circle intercept procedure through experts of equivalent level, labels the images with the LabelMe v5.1.1 software and uploads them to this dataset for image acquisition. The module contains 2000 effective images of grain size numbers and their labeling files, which are used as the training set of the contour extraction network module.The contour extraction network module is mainly for image preprocessing and grain boundaries contour extraction. One of the most critical steps is the training of the model, which is specifically divided into the following substeps: (1) Calibrate the original data of image size 2048 × 1536 with LabelMe, mark the visible boundary and fill the prolongable boundary. Name the images with id_magnification used_level.png See . (2) TheStart the test after the model is trained. Enter a metallographic image of size 2048 × 1536. The program first segments the image, runs the contour extraction model for model extraction, and finally splices the segmented images, and processes the overlapping part with the mean value. In this way, a complete contour extraction image will be obtained. Both model training and prediction are carried out in an end-to-end manner. Extraction of contours by sections greatly reduces the complexity of model training and improves the running speed of the model. The formula for dealing with overlapping parts is as follows:As for the visible intersection count statistics module, the binary image of edge contour output by the contour extraction network is input, and based on the three-circle truncation method, the edge contour line is judged to intersect the circle, and the position of the pixel where the intersection point is recorded, so as to conduct preliminary statistics of the truncation point, so the number of visible cut-off points P is calculated., finally, present As for the hidden intersection count statistics module, the binary image of the grain boundary contour output by the contour extraction network is input, and the intersection count of the pearlite area is estimated based on confidence. The calculation of grain size number is a description of the overall ferrite properties. It may be assumed that the size of each grain boundary in a ferrite section is approximately normally distributed after the hidden boundary is delineated. The specific steps are as follows: After the grain boundary segmentation of the first part, we obtain a set of grain boundary sizes The automatic rating module accumulates the visible and hidden intersection count to obtain the final intersection count and calculates the grain size number by Formula (1).The model architecture is shown in The training datasets in this paper are all about hot rolled ribbed steel bar samples (HRB400). A trained model is used to test the grain sizes of four different grades of steel, namely SWRCH6A , HRB400E (with a carbon content ≤ 0.23%), 35K (with a carbon content of 0.32%) and S45C (with a carbon content of 0.45%). With different chemical compositions, they are all of a ferrite–pearlite microstructure. The two-phase ratio of samples with different carbon contents is different. Generally, the higher the carbon content, the higher the pearlite ratio.The original image size of the datasets used in this paper is 2048 × 1536, and the sub-images of 256 × 256 are cut out with a step size of 128, which fall into the training dataset, verification dataset and test dataset, accounting for 70%, 20% and 10% of the total data respectively. On top of that, the grain size number of three types of steels for which no training data collected are tested.During model training, the authors adjust the number of iteration rounds, learning rate, batch size and other parameters. The results of grain boundary segmentation of various types of steels are shown in A three-circle intercept procedure is used to count and calculate the grain boundary intersections in the segmented grain boundary images. Three concentric, isometric circles with a total perimeter of 500 mm are used to form a measurement grid, and the point of intersection between the grid and the grain boundary is the intersection. The number of visible intersections in the ferrite microstructure is obtained by counting the binary images of the edge contour with the three-circle intercept procedure. The intersections of the four materials are shown in In the end, the automatic rating module integrates the visible intersections and hidden intersections to output the rating results and the labeled image of the three-circle intercept procedure. Then, calculate the average grain size number of the samples according to Formula (1). The rating results of the grain size number of the four samples are shown in ;th sample of class The average error and accuracy are defined as follows:This paper is based on the model training, prediction and data processing of metallographic images, which inevitably relies on the quality of images. An image will be more representative in characteristics if it has fewer interference items and richer data, and as a result, the model training effect will be better. The original metallographs do have such defects as scratches, foreign contaminants and holes, and they all are interference items for the training and prediction of the deep learning model. In this paper, two methods are mainly used to eliminate the negative impact of these interference items on the model as far as possible. Method 1: accurate data calibration. Scratches, foreign contaminants and holes are different from grain boundaries in form. For example, scratches are mainly long straight lines, while foreign contaminants and holes are mostly points rather than curves. They are obviously different from the grain boundaries that we need to identify. In the preliminary data calibration work, we arranged senior metallographic analysis experts to identify and calibrate the grain boundaries and would not mark these interference items as inclusions. The accurate calibration ensures that, in subsequent training and identification, the model algorithm is unlikely to identify these interference items in the verification sample as grain boundaries. Method 2: Identification through the algorithm. As mentioned earlier, interference items differ from grain boundaries in size, grayscale or shape, and we can avoid misjudgment as far as possible by identifying these differences through the algorithm. For example, in terms of size, grain boundaries are curve segments, so we can directly use the algorithm to delete particles whose length and width are less than a certain value . In terms of grayscale, the grain boundaries are gray or nearly black. Items with lighter or darker colors are generally foreign contaminants, holes, corrosive water stains, or other interference items. We set a grayscale threshold in the algorithm, and this can also exclude interference items to some extent.This testing process is an approximate, simple and fast method, suitable for mass production inspection in factories, and cannot be used for accurate detection of grain size. If the true and accurate grain size of the sample is required, it is also necessary to detect the grain size of pearlite (the size of pearlite colonies) besides that of ferrite. In this case, it is necessary to use a scanning electron microscope to obtain a metallographic image with higher resolution, which can clearly display the boundaries of pearlite colonies, and thus the grain size of pearlite can be tested. As we know, the pearlite microstructure is in colonies due to different orientations of ferrite–cementite lamellae, and the boundaries between microstructures of different orientations can be regarded as the grain boundaries of pearlite colonies. Unlike solid boundaries such as ferrite grain boundaries, such boundaries are not solid boundaries formed by lines or specific grayscales or colors, but virtual boundaries formed only due to different orientations of lamellae. The areas on both sides of such virtual boundaries have no difference, except the difference in the orientations of lamellae. Unlike the hidden boundary described earlier as well, this virtual boundary is objectively present and is not visible to the human eyes simply because of the low resolution in the optical images. Such boundaries are easy to identify for man, but difficult for computer. We are studying this aspect and have achieved good results, which will be presented in the next paper.This paper presents a model for automatic detection and quantitative analysis of the grain size number of ferrite–pearlite two-phase microstructure. A U-Net neural network model is designed to segment ferrite grain boundaries. In view of the hidden grain boundaries in pearlite microstructure, the number of hidden grain boundaries can be inferred based on the confidence of average grain size. The grain size number is rated using the three-circle intercept procedure. The results show that the accuracy of this procedure is greater than 90% in the grain boundary segmentation for the four types of ferrite–pearlite two-phase microstructure samples. All of these results deviate from those calculated by experts using the manual intercept procedure by less than Grade 0.5, the allowable detection error specified in the standard. In addition, the detection time is shortened to 2 s from 30 min consumed in the manual intercept procedure, greatly improving the detection efficiency and reducing labor intensity. Accordingly, the method proposed in this paper can be effectively used for the automatic rating of grain size number of ferrite–pearlite microstructure. The results obtained can be applied in steelworks laboratories to control the grain size of rolled ferrite–pearlitic steel.


In order to solve the problem of long video dependence and the difficulty of fine-grained feature extraction in the video behavior recognition of personnel sleeping at a security-monitored scene, this paper proposes a time-series convolution-network-based sleeping behavior recognition algorithm suitable for monitoring data. ResNet50 is selected as the backbone network, and the self-attention coding layer is used to extract rich contextual semantic information; then, a segment-level feature fusion module is constructed to enhance the effective transmission of important information in the segment feature sequence on the network, and the long-term memory network is used to model the entire video in the time dimension to improve behavior detection ability. This paper constructs a data set of sleeping behavior under security monitoring, and the two behaviors contain about 2800 single-person target videos. The experimental results show that the detection accuracy of the network model in this paper is significantly improved on the sleeping post data set, up to 6.69% higher than the benchmark network. Compared with other network models, the performance of the algorithm in this paper has improved to different degrees and has good application value. In recent years, with the rapid growth of video surveillance data and the rapid development of artificial intelligence, big data, the Internet of Things and other technologies, society’s emphasis on public safety has gradually deepened. Intelligent perception technology based on video surveillance has become important. Intelligent sensing technology based on video surveillance has become an important direction of the development of security monitoring , and is Human target detection and behavior recognition are the most common basic problems in security tasks. Because of the subjective initiative and non-rigid characteristics of humans, this is a complex research topic in the field of computer vision. Human detection detects and locates a human body from video, and behavior recognition identifies and analyzes human behavior. Human body detection is a basic component of intelligent processing systems, such as security and transportation. The real-time and accurate detection of human body targets is also the basis for realizing human abnormal behavior analysis and other visual tasks. Behavior recognition is an important part of personnel intelligent management systems. The automatic and real-time intelligent monitoring of personnel using video can greatly improve personnel management levels. In practical work, sleeping on duty can potentially lead to safety hazards, especially when sleeping on duty at night causes safety accidents. By using a sleeping behavior recognition algorithm, the sleeping condition of the personnel on duty can be detected, and corresponding preventive measures can be taken to reduce the occurrence of accidents. The adoption of a sleeping behavior recognition algorithm can digitally display the working state of on-duty personnel, improve supervision efficiency, and avoid dangerous behaviors caused by the subjective factors of the on-duty personnel not being detected and stopped in time, which will lead to serious consequences.In this paper, we propose a long video classification network incorporating a self-attention mechanism. This method is divided into two stages, feature extraction and feature fusion enhancement, which can effectively realize the transmission of important information in the segment feature sequence. In the feature extraction stage of a CNN , we first added the self-attention mechanism module to obtain the fine-grained features between long video sequences, solving the problems caused by difficulty in extracting the fine-grained features of people’s sleeping behaviors. In the feature fusion stage, the segmentation strategy module was added to effectively solve the problem of long-distance dependence of spatiotemporal information in long video feature fusion. The experimental results show that this algorithm improves the transmission ability of the network segment feature sequence, further improves the CNN-LSTM  network’s detection accuracy of sleeping behavior, and has high practical application value.-An overview of methods, models and algorithms used in personnel behavior recognition based on deep learning;-A sleeping post data set built under the monitoring situation for the training of a sleeping personnel post task network model;-For the task of recognizing sleeping personnel behavior, a sleeping behavior recognition algorithm based on a self-attention time convolution network for monitoring data is proposed: 1. In the feature extraction stage of a CNN network, the self-attention mechanism module is added to obtain the fine-grained features of the image, so that the feature extraction network pays more attention to the movement features around the people, solving the problems caused by difficulty in extracting the fine-grained features of people’s sleeping behaviors. 2. In the feature fusion stage, a video segmentation strategy is proposed, which uses segment-level features to stack into video-level features to effectively solve the problem of long-distance dependence of spatiotemporal information in long video feature fusion.The contributions of this work are summarized as follows:The rest of this paper is organized as follows: The feature extraction of the early traditional image processing method  mainly rAlthough a great deal of research has been carried out in the field of video behavior recognition, there are still great challenges in the recognition of drowsy abnormal behavior in the monitoring environment. The difference between sleeping behavior and general behavior is that sleeping behavior lasts for a long time, sleepers produce less motion information within a certain period of time, and that sleeping behavior is highly similar to the representational information and spatiotemporal characteristics of daily office behavior. Additionally, because the monitoring devices are often far away, it can be difficult to clearly observe the state of the face, which makes it difficult to recognize sleeping behavior through face judgment.For sleep scenarios, most sleep detection studies ,28,29 haPeople’s understanding of motion video comprehensively describes their motion information. Understanding the feature association between video frames is essential to improve the accuracy of video motion recognition. In a time series, not all information is equally effective. The previous information can be used to help understand the subsequent content. LSTM is widely used in the research of human behavior recognition because it can process long time series data and has good memory ability. Therefore, we chose LSTM as our final decoding module. The excellent structure of LSTM can ensure that the sequence path from the previous unit to the current unit still exists, meaning that the network can provide a large amount of long-term information compared with other RNN networks. However, in fact, LSTM can only remember limited distance information at most. For longer-distance information, LSTM has a general effect and is obviously insufficient in image feature extraction. As a special case of an attention mechanism , self-atThe self-attention score was calculated and aggregated according to Equations (1)–(3) to obtain the output result of the self-attention mechanism represented by a feature vector. Assuming that the input feature is g(·) and y(·) are linear transformation operations (here is a convolution operation of 1 × 1); D stands for the number of output channels, except for the regulation of D, so that the training process has a more stable gradient; score is a self-attention score of softmax; and conv(·) means 1 × 1 convolution. Finally, we weighed the weight and the corresponding value and obtained the self-attention score of 8 ×1 × 1 × 1024.Among them: In video classification, the length of the video sequence will significantly impact the network performance. A convolutional network is powerless to model a long-range time structure, mainly because it uses only a single-frame image stack or only one frame in a short segment. Hence, access to the space–time context is limited. Although a temporal segment network (TSN)  effectivAiming at the problems in TSN , we propWhen the segment-level feature fusion module divided the video sequence into multiple segments, we used 4 frames, 8 frames and 16 frames as the segment length of the entire video sequence. If the segment length is too short, the similar feature vectors in the entire sequence will still have redundancy, resulting in the advantages of the segmentation strategy not being fully reflected; however, a segment that is too long will not only lead to an exponential increase in the amount of computation, it will also cause the loss of some important features. After our experiments, we finally chose 8 frames as the video sequence partition interval. Although we increased a part of the computation amount during segment division, we selected most features before entering LSTM, which reduced the number of feature vectors entering the LSTM layer, so as to not reduce the efficiency of the overall network.The picture frame is output as a feature vector of 1 × 1024 through the CNN encoding layer. We divided a feature sequence of the completed video into equal-length segment sequences and then calculated the distance between all feature vectors and other feature vectors in each segment:The video classification network model proposed in this paper is based on CNN-LSTM and consists of two modules: a CNN-based encoder and an LSTM-based decoder. The overall structure design of the network is shown in −4. A total of 100 epochs were iterated. The early stop method was used to avoid overfitting.This paper implemented the computing environment configuration as follows: hardware , software , and software tools . The optimizer used for network training was the Adam optimizer. The initial learning rate was 1 × 10In this paper, a long video action behavior recognition algorithm based on the video segmentation strategy was studied, mainly focusing on detecting abnormal sleepiness in video surveillance. For this study, the current public data set could not meet the requirements well. Due to the existence of the long tail effect, although the public data set also contains common abnormal behaviors, the amount of data was very small, and it could not be well-trained for the abnormal sleepy behaviors covered in this paper. Although the method in this paper was to identify the sleeping behavior of the monitored individuals from a single perspective, considering the discreteness of the data set and the number of data samples, we used monitoring videos from different perspectives to build the data set. To this end, three surveillance cameras were installed in three corners of the laboratory to build a data acquisition environment to obtain video of sleepy behavior, and a video data set was built. The environment under video surveillance is more complex, with many personnel targets. As an independent individual, each person has different characteristics. If the original image is used as the training set, it will inevitably cause a lot of feature redundancy and reduce the network recognition ability. Therefore, in establishing a data set, capturing the suspected sleepy people from the video for initial screening is necessary. In this paper, the following methods were used to capture specific targets. The data set establishment process included six steps, as shown in The description of sleeping behavior under the surveillance video is usually a single person’s behavior state with small action amplitude or relatively static behavior for a long time. When performing sleep recognition analysis, we should pay more attention to the changes in a single person’s front and back frames. Firstly, the YOLOv4 target detection network was used to detect and track the original video image, and the single-person target frame was extracted to form a video image sequence; considering the high cost of optical flow calculation, the inter-frame difference method was used to obtain the contour of the moving object from two consecutive frames of the video image sequence. When the object moved in a large range, there were obvious differences between the two adjacent frames. The expansion and erosion algorithm was used for morphological processing to enhance and eliminate noise. The absolute value of the pixel value difference was obtained at the corresponding position of the image sequence; the median value and average value of the pixel were calculated; it was judged whether they were less than a certain threshold; and then the image sequence that belonged to the suspected sleeper was extracted. In reality, due to the long distance between the surveillance cameras, people who have not worked for a long time will also be misjudged as likely to sleep, so it was necessary to manually filter the data into two categories, sleeping behavior and normal behavior, to complete the construction of the video data set.Depth learning methods usually improve the accuracy with the increase in training data. For video action recognition, this means that large-scale labeled data sets are needed to train effective models. Due to the difficulty of self-collecting data, the number of sleepy video data sets under monitoring was small, and the self-collected sample data only contained 306 video clips. Therefore, this paper considered the method of changing the frame and taking the time of video clips to expand the data set to increase the sequence of sleepy images. In this paper, video images were taken at intervals of 5 frames, 30 frames and 3 different forms of video samples of their mixed frame combinations as data sets, and sent to the CNN-LSTM network for training. The experimental results are shown in It can be seen from TP represents the number of real categories that are actually real and predicted; TN refers to the number of error categories predicted to be error categories; FP represents the number of actual error categories that are predicted to be real categories; and FN indicates the number of correct categories that are predicted to be wrong categories.The behavior recognition network model accuracy evaluation index uses the accuracy rate to evaluate the video classification network. The calculation formula is as follows:In the sleep data set, all video samples contained only a single target. The sleep state and other states were selected as the classification data set. The two behaviors included about 2800 videos, of which 1988 videos were used as the training set, and 426 videos were used as the test set. The distribution of experimental data is shown in We conducted training models for both the reference network and the improved video classification network and output the corresponding classification accuracy curve and loss decline curve during the training process, as shown in As shown in In order to see the impact of each module on the accuracy of the network model, we separately performed some ablation experiments for each module we proposed when the segmentation strategy interval was eight frames  Introducing the self-attention mechanism into the network coding layer and combining it with the ResNet50 structure to enhance the feature relationship between image sequences was effective, which helped the network fully capture fine-grained local features from the encoded image features to enhance the feature relationship between image sequences. (2) Comparing the Euclidean distance between feature vectors passing through the coding layer can be used to calculate the similarity between features and fuse the most representative and the most distinct features. This method makes the network pay more attention to the differences between image sequences, which can further improve the accuracy of sleep behavior recognition.Compared with other behavior recognition, sleep behavior recognition is a more challenging behavioral recognition task. Currently, no effective method has been found to solve the problem of sleep behavior recognition. Because of its long behavior duration, less movement information is generated in time and space, and the problem of sleep posture may be similar to daily office representative information and space–time characteristics. Thus, the network needs to have a more sophisticated feature difference capture capability to solve the problem of sleep behavior recognition. In cognitive neuroscience, attention is an indispensable complex cognitive function of human beings. It is reflected that the human brain can intentionally or unintentionally select a small piece of useful information from a large amount of external input information to focus on processing and ignoring other information. The purpose of introducing the self-attention mechanism into the sleep recognition network is to enable the network to have the ability to self-judge the importance of features in the process of feature learning. Then, the two coded features with significant differences are combined into segment-level features instead of the original random sampling method in the segment. This effectively enhances the network’s ability to extract global features and increases the recognition accuracy of the sleep data set. The action recognition method of the long video sequence based on video segmentation feature extraction and fusion proposed in this paper has a specific reference value for action recognition under long video sequences. It can provide the technical basis for the intelligent detection of personnel sleeping behavior under security monitoring.