Introduction: The challenge of navigating a Mobile robot in dynamic environments has grasped significant attention in recent years. Despite the available techniques, there is still a need for efficient and reliable approaches that can address the challenges of real-time near optimal navigation and collision avoidance.Methods: This paper proposes a novel Log-concave Model Predictive Controller (MPC) algorithm that addresses these challenges by utilizing a unique formulation of cost functions and dynamic constraints, as well as a convergence criterion based on Lyapunov stability theory. The proposed approach is mapped onto a novel recurrent neural network (RNN) structure and compared with the CVXOPT optimization tool. The key contribution of this study is the combination of neural networks with model predictive controller to solve optimal control problems locally near the robot, which offers several advantages, including computational efficiency and the ability to handle nonlinear and complex systems.Results: The major findings of this study include the successful implementation and evaluation of the proposed algorithm, which outperforms other methods such as RRT, A-Star, and LQ-MPC in terms of reliability and speed. This approach has the potential to facilitate real-time navigation of mobile robots in dynamic environments and ensure a feasible solution for the proposed constrained-optimization problem. Neural networks and Model Predictive Controller (MPC) are two powerful techniques in robotics, especially in the path planning of mobile robots. Neural networks are computational algorithms, mimicking human brain functions , while MDepending on the specific requirements, engineers may find both neural networks and MPC to be highly useful. In fact, a combination of these methods can provide robotics experts with even greater capabilities to enhance the efficacy and reliability of path planning .In the realm of path planning for mobile robots, it is imperative to not overlook the significance of model-free techniques, even while capitalizing on model-based approaches such as MPCs. Although model-free control methods, exemplified by Q-learning and its derivatives, have been extensively studied in mobile robot navigation , their cThe literature review reveals that most of the methods in recent research have utilized neural networks to predict the environment for robot navigation . HoweverIn a recent study a learning-based nonlinear model predictive controller (LB-NMPC) algorithm is presented to improve vision-based mobile robot path-tracking in challenging outdoor environments . This apWhile most of the existing literature focuses on using neural networks to predict the environment, our work emphasizes utilizing neural networks to solve optimal solutions locally near the robot. This approach confers advantages such as increased accuracy and efficacy in the navigation of robotic systems.Other recent research has also been conducted regarding path planning and navigation for mobile robots. For example, a study proposed an improved deep reinforcement learning algorithm for path planning in large-scale dynamic environments, considering the avoidance of dynamic obstacles . AnotherOn the other hand, the process of identifying the most efficient trajectory for a mobile robot is inherently characterized by non-linearity and occasionally non-convexity, thereby restricting the applicability of standard convex optimization tools in the context of MPCs applications. In this regard, a MPC that is based on neural networks (Neuro-MPC) can prove to be particularly advantageous when navigating complex environments in which dynamic obstacles cannot be readily modeled using convex optimization tools .Although the combined method mentioned above offers numerous advantages, it is crucial not to overlook the importance of system stability. This issue cannot be neglected and must be taken into consideration. The stability of Neuro_MPC is a critical issue in path planning because instability can lead to unexpected behavior of the autonomous system, which may cause accidents or damage to the environment . SeveralIn order to tackle the challenges at hand, we have developed a novel heuristic log-concave MPC navigation problem that takes into account dynamic and kinematic constraints, and guarantees convergence based on Lyapunov stability theory. To solve this problem, we have designed a new structure for a recurrent neural network that is inspired by the NN constrained formulation presented in , and utiOur exploration revolves around a three-wheeled omnidirectional robot that uses a differential drive scheme to control its motion. This distinctive robot architecture confers greater dexterity and accuracy in traversing intricate environments, particularly when confronted with mobile obstacles. Our work employs the built-in tracking controller, which is based on the controller devised by This paper is organized into several sections to provide a comprehensive overview of our approach. In The general navigation system for robots involves various factors such as the positions of the target and the robot, the environment’s characteristics and arrangements, and the constraints and objectives.  block to receive updated information on the position classified data and evaluate the next state policy cost of the robot’s neighborhood, considering a conservative collision avoidance constraint compatible with finite state prediction steps. By combining local and global information, our proposed path planning system offers a more comprehensive and effective approach to robot navigation.In order to assess the effectiveness of our navigation strategy, we utilized Python 3.10 to simulate three randomly moving obstacles, as well as a real three-wheeled omnidirectional mobile robot (OMR) dynamic model with a built-in tracking controller introduced by . Our goaBy and large, we used the EKF algorithm for obstacle tracking and estimation. The EKF algorithm is used to estimate the position and velocity of obstacles and the robot in the environment, which are utilized as inputs to the MPC algorithm to generate control actions for the robot. The EKF algorithm considered the position and velocity of the obstacles and predicted their next position by predicting one step ahead.The MPC algorithm was responsible for generating control actions that steer the robot towards the target while avoiding collisions with obstacles. The MPC algorithm used the estimated positions and velocities of obstacles obtained from the EKF algorithm as inputs to generate control actions for the robot. The cost function used in the MPC algorithm was designed to balance the tradeoff between achieving the desired goal and avoiding obstacles while minimizing energy consumption.The RNN structure is designed as a local optimization solver for the nonlinear MPC problem. Specifically, the RNN is used to estimate the system dynamics and cost function, which are then used as inputs to the MPC algorithm to solve the optimal control problem. The RNN is trained to minimize the cost function in the Lagrangian form by adjusting the input control to the system at each time step, such that the constraints are satisfied and the state of the system approaches the desired reference trajectory.This section provides a comprehensive description of the preprocessing and analysis of the sensor data utilized in our simulation study. In consideration of the EKF converging process and RNN learning and training time samples, we utilized 70 sampling epochs as the training dataset, using data recursively, and the remaining epochs for testing.The sensor data is simulated as flag data, which detects obstacles from a distance of 0.4 m. If obstacles are located in front of the sensor, the flag is set to 1, otherwise, it is set to 0.To generate the input data for the learning algorithm, we used a camera mounted on the ceiling of the room to detect the position and color of obstacles in the environment. The position of the robot and obstacles were labeled as global positions, and the obstacles were represented as binary values, with 1 indicating the presence of an obstacle and 0 indicating no obstacle at that position. We also added normal random noise to the positions of the robot and obstacles to simulate the measuring noise in the sensor data.To analyze the sensor data, we first pre-processed the data to remove noise and outliers and normalized the data to ensure that the learning algorithm could effectively learn from the data. We also used data augmentation techniques to increase the size of the dataset and improve the generalization ability of the learning algorithm.Next, we used the Extended Kalman Filter (EKF) algorithm to estimate the position and velocity of obstacles in the environment and the robot. The EKF algorithm considered the position and velocity of the obstacles and estimated their next position by predicting one step ahead. Obstacles were designed with random velocity and shape to simulate a realistic environment. If any new obstacles were detected, a new EKF was started to estimate its position. If the predicted location and sensor data were not compatible, the sensor data was given priority, and the robot changed its direction by 45° in each step while the sensor flag was set to zero to show the lack of obstacles nearby. Finally, we used the pre-processed sensor data as input to the learning algorithm and trained the algorithm to predict the position and velocity of obstacles in the environment.To simulate realistic dynamic behavior of a mobile robot, a real three-wheeled OMR Dynamic model parameters and internal tracking control law, given by  are utilThe inertial matrix, the Coriolis and centrifugal force vector, and the input matrix, defined as follows:The parameters of the model are stated in This paper employs m+1 Kalman filters concurrently to forecast the future states of m obstacles, as well as the robot’s position and velocity, based on the mean square error of the estimation. The Kalman Filter Parameter estimation laws are derived from , and theThe estimated future states of the robot are utilized to predict the future screen of the environment. Moreover, to predict future states of the robot, a Kalman filter is implemented to estimate the parameters of an Auto-Regressive Exogenous Model, which is considered as the dynamic constraint of the robot in the proposed MPC navigator.This section addresses a common challenge in obstacle avoidance, which involves dynamic and kinematic constraints within a log-convex MPC framework. In order to tackle this issue, we have identified several key rules that are typically applied in navigation problems. These rules have been translated into mathematical expressions that can be incorporated into the MPC cost function and constraints.Rule one emphasizes the importance of taking the shortest route possible, while rule two highlights the value of maintaining a safe distance from obstacles. Rule three prioritizes the use of sensor information over image processing for obstacle detection, and rule four stresses the need for accurate tracing at all times. Rule five reminds us to adhere to the maximum allowable speed, and rule six emphasizes the importance of ensuring system stability.To incorporate these rules into our MPC framework, we have developed a set of linear constraints and kernel functions that allow us to evaluate their relative importance on a scale of zero to one. Rules one to five are incorporated into a constrained MPC structure, which is defined as follows:d indicates maximum possible movement in the Environment toward the (x-y) axis. In addition, the maximum possible rotation angle is shown by Vmax denotes the maximum possible velocity for the robot, according to the environment and robot dynamic. Uk is the control law, i.e., the generated path. Flag is 1 if the built-in sensors detect obstacles around the robot, otherwise, it is zero, to deal with uncertainties and possible faults of environment predictions. This item is considered to avoid the robot collision with obstacles, in the KF failure situations.Where The cost function used in this study is a key component of the model predictive controller (MPC) algorithm that controls the navigation of the robot. It is designed to balance the tradeoff between achieving the desired goal  and avoiding obstacles while minimizing the energy consumption of the robot. The position cost function is given by the term This paper utilizes ARMAX model to estimate velocity and position of the robot as follows: k denotes the position and velocity of the robot at kth sample time, and Where YConsidering constant control input in each sample time, we define N-horizon prediction of states vector as follows:Pk is the prediction of N next positions of the robot.Where Doi= T denotes to the ith obstacle’s position and velocity vector in the environment at the global coordination. Obstacles’ states predications are calculated below:Suppose that To show the constraint of the robot and obstacles collision avoidance, we define the below criteria:Where Due to possible prediction faults and uncertainties, the importance of checking robot obstacle detection sensors increases, so we consider a Flag to detect an obstacle. Flag = 1 indicates an obstacle is nearby the robot, and Flag = 0 reveals the lack of obstacle around the robot.This inequality can also be considered as a fitness neighborhood searching function, instead of intending it as another constraint of the MPC problem. This substitution helps to preserve convexity of the problem, when convex optimization solver is utilized.Kinematic constraints should be regarded in the navigation strategy. This study considers a mere velocity constraint.This section provides an error feedback control law, F is discrete-time error-feedback gain In order to ensure stability in a navigation system, it is necessary to determine the criterion for stability guarantee. This can be achieved by calculating the derivative of the defined Lyapunov function, which should be negative. By simplifying the Lyapunov inequality constraint of stability, it is possible to determine whether the navigation system is stable.Stability in navigation is crucial for ensuring that a robot can reach its intended destination without any collisions. To simplify the stability criterion, a linear approximation of the cost function is used, taking into account the limited arguments of the kernel functions, which are restricted to values of 0 and 1. By following these steps, it is possible to ensure that the navigation system is stable and reliable, allowing for safe and efficient navigation.This section is assigned to map the newly defined optimization problem into the searching-based recurrent Neural Network structure. To this aim, consider a general Lagrangian constrained optimization problem as follows :Lx1,x2,A new and innovative RNN structure has been proposed to find a near-optimal solution in possible movement directions. This structure is made up of newly designed Gaussian spiking neurons that construct the cost function criteria concept in their membrane function. The synaptic neurons are defined in such a way that the network can search for the best position.The main idea behind this proposed structure is to solve the Lagrangian problem by considering synaptic current as a term that can indicate the weighted constraints. This synaptic current is then inputted to the synaptic neurons.After analyzing the low-risk position corresponding to the minimum membrane potential, the activation function generates the desired path according to the calculated feedback gain corresponding to the low-risk possible location. In other words, the minimum Vn indices indicate the path error feedback indices, and consequently, the desired path is calculated.Sij is a Boolean value that represents the state of a grid cell in the global screen. The screen is divided into a grid based on the scale of the robot’s dimensions, treating the robot as a single point in the encoded matrix of the environment. The values of xr and yr represent the robot’s position in the (x-y) coordinate system. The variable Vn in the equation represents the cost of each state in the neighborhood of the robot, as well as the cost of searching around it, considering one-step movements and calculating the cost of the distance from the target. Sij is used to increase or decrease the cost if an obstacle is present or absent at that position. The first term of Vn searches for low-risk positions around the robot, while the next term evaluates the risk of one-step movements.The proposed network algorithm is defined in Eq. This searching structure is designed with the aim of reducing map size dependency and consequently its computational cost. In is the input synaptic current on the neuron n and Nv is the neuron membrane function, which indicates the proposed cost function and 1 step prediction of moving environment screen. The combination of the searching-based idea with the reservoir structure can reduce sensitivity of the proposed navigator to uncertainties and improve its robustness.Wij) are updated based on minimizing Vn, as an indicator of the th Lagrangian function of the proposed algorithm.The input neurons and the size of proposed network can be considered constant and relevant to the steps of prediction horizon. Synaptic weights . The robot movement path is simulated under the following conditions:The global screen of the environment, including the movement path of the robot and obstacles are shown in It is evident that generating a path using RRT can be time-consuming, especially if a unique solution is obtainable in this structure. To better understand this issue, we have provided a table below that compares our proposed method with RRT in terms of distance versus time. The simulation screen size is 20*20 with a scale of 0.45 m, and the arrangement of obstacles differs in at least five similar simulation examples. The target position is located at  in FigurOur proposed method offers a more efficient and effective solution for navigation in a simulated environment. With our method, the robot can navigate through the obstacles and reach the target in a shorter amount of time compared to RRT. This not only saves time but also reduces the risk of errors and increases the overall success rate of the navigation process.It is evident that generating a path using RRT can be time-consuming, especially if a unique solution is obtainable in this structure. To better understand this issue, we have compared our proposed method with RRT in terms of distance versus time in Our proposed method offers a more efficient and effective solution for navigation in a simulated environment. With our method, the robot can navigate through the obstacles and reach the target in a shorter amount of time compared to RRT. This not only saves time but also reduces the risk of errors and increases the overall success rate of the navigation process.The proposed RNN-Log-MPC algorithm presents a notable advantage by utilizing both sensor and camera data to diminish the sensitivity of the navigation system to momentary data loss or prediction error by EKF. Nonetheless, this does not eliminate the exigency to tackle potential issues such as prediction errors, sensor disconnection, and lack of accuracy in the command follower controller. Therefore, a conservative approach to adjusting and evaluating restrictions may be deemed necessary, which could constrain optimization, particularly in unfamiliar environments where speed changes occur, and there is a possibility of collision risk.Moreover, the precision of obstacle detection, which is processed via a central server and communicated wirelessly to the robot’s internal controller, can also influence the system’s performance. Hence, it is crucial to scrutinize the algorithm in various scenarios to identify potential errors and devise solutions to rectify them. Constant monitoring of the system can also enhance the accuracy of performance and reliability of the navigation system.This study proposes a groundbreaking navigation algorithm that employs a log-concave strategy and a visual-based model predictive controller (MPC) framework. The tactic is innately near-optimal and restricted, and it is tackled by employing two optimization resources, namely, a newly proposed network configuration and the CVXOPT optimization library. The suggested recurrent neural network (RNN) architecture adeptly resolves diverse constrained navigation predicaments. The constraints, which include the velocities of the robot and the nearby obstacles, are given careful consideration to enhance the reliability of the navigation system. Our simulations demonstrate promising results, but further research is imperative to scrutinize the applicability of our methodology in pragmatic settings or using tangible mechanisms such as Gazebo or VREP to validate our approach and provide additional insights.This investigation furnishes a resolution to the optimal visual navigation technique, where the structure size remains constant in various environments with different numbers of obstacles, by utilizing a window-based searching approach. Nonetheless, the Extended Kalman Filter, which is utilized to localize objects, is a highly favored stochastic method in this area, but it operates based on the linearization method. Consequently, there is still a concern about its failing events, which can trigger low-accurate predictions of the environment.To summarize, our proposed RNN-based optimization methodology for circumventing obstacles showcases auspicious outcomes and harbours the potentiality of being expanded to more intricate scenarios and environments. For instance, our methodology may be implemented in pragmatic situations such as self-governing motoring, wherein the robot necessitates traversing through a multifarious and dynamic milieu while evading hindrances. Furthermore, our approach has the capacity to amalgamate with alternative state estimation and prognostication techniques to advance the performance and resilience of the system. We are of the conviction that our study lays a robust groundwork for ensuing research in the establishment of RNN-based techniques for circumventing obstacles in autonomous robots.


Simultaneous localization and mapping (SLAM) algorithms are widely applied in fields such as autonomous driving and target tracking. However, the effect of moving objects on localization and mapping remains a challenge in natural dynamic scenarios. To overcome this challenge, this paper proposes an algorithm for dynamic point cloud detection that fuses laser and visual identification data, the multi-stage moving object identification algorithm (MoTI). The MoTI algorithm consists of two stages: rough processing and precise processing. In the rough processing stage, a statistical method is employed to preliminarily detect dynamic points based on the range image error of the point cloud. In the precise processing stage, the radius search strategy is used to statistically test the nearest neighbor points. Next, visual identification information and point cloud registration results are fused using a method of statistics and information weighting to construct a probability model for identifying whether a point cloud cluster originates from a moving object. The algorithm is integrated into the front-end of the LOAM system, which significantly improves the localization accuracy. The MoTI algorithm is evaluated on an actual indoor dynamic environment and several KITTI datasets, and the results demonstrate its ability to accurately detect dynamic targets in the background and improve the localization accuracy of the robot. SLAM is the basis for robots to engage in exploratory work in unknown environments, and it is also an essential manifestation of robot autonomy. In standard SLAM algorithms, the assumption of a static environment is very typical, while in more cases, the robot is exposed to a dynamic environment. Moving objects in the background will act as noise, which may degrade the sensors and inhibit the system’s positional estimation robustness. Furthermore, the environment perception capability is a facilitator for later applications in service robots or autonomous driving. Therefore, detecting and eliminating dynamic obstacles is vital for improving the effectiveness of localization and map construction.At present, conventional methods for target detection include vision-based , laser-bSingle-sensor systems may result in degraded map quality due to limited sensing capabilities. In contrast, multi-sensor information fusion acquires different feature information from different sensors, which enriches the measurement information and is helpful for environment perception and state estimation. However, there are still several challenges that have to be overcome in the system design process. On the one hand, since laser point cloud data are sparse and image data are a complete description of a certain angle range, the difference in information sources leads to more difficulty in data fusion. On the other hand, a proper judgment mechanism has to be introduced to prevent point clouds from moving objects from being misjudged as static points.Therefore, a suitable multi-sensor fusion system combined with supervisor learning for point cloud processing can be designed to effectively detect dynamic features. In this paper, an algorithm for dynamic point cloud detection that fuses laser and visual data is proposed. By fusing the detection information of YOLOv5 and point cloud data, we design a judgment mechanism for moving point clouds based on statistical information through multiple stages of point cloud processing. In this manner, we achieve the detection and separation of dynamic point clouds. Static point clouds are used as input for SLAM to improve the accuracy of localization and point cloud alignment. The main contributions of this study are as follows: (1) A point cloud processing method was proposed based on two-stage processing to achieve the detection and marking of point clouds on moving objects. (2) We apply the MoTI algorithm to the front end of LOAM to achieve a complete marking of the point cloud on moving objects. Based on this, we improve the localization accuracy and mapping effect of the system. (3) Simulation and indoor dynamic scenario experiments were carried out, respectively, and the results show good performance in identifying moving targets and robot localization.The paper is organized as follows: SLAM algorithms for dynamic scenes are mainly divided into vision-based and laser-based processing approaches. The current mainstream methods mostly use vision or laser information combined with deep learning methods to supply semantic information to the environment. The literature on fusing laser and visual identification information in SLAM for moving target detection is limited.Some SLAM algorithms based on static assumptions of the environment are also robust in dynamic environments, such as ORB-SLAM , which gThe methods of deep-learning can effectively identify common dynamic targets in the environment. Fan and Zhang et al. proposed Blitz-SLAM , which cIn , RGB imaCompared with vision-based methods, laser data can describe the geometric information of space more accurately. Laser-based moving object processing includes dynamic target detection  and tracLim et al.  proposedThe existence of moving objects in the environment causes the point cloud matching process to unavoidably substitute dynamic points into the iterative computation, which leads to a bias in the state estimation. Therefore, this paper proposes the MoTI algorithm that removes dynamic objects based on the fusion of lidar and visual identification information during two stages, which is applied to point cloud processing in low-dynamic scenarios. In order to improve the computational efficiency, we complete the operations of ground point cloud rejection, region of interest selection, point cloud clustering, and image detection data encapsulation before the rough processing stage. The moving object removal module is a major part of the MoTI algorithm, combining statistical ideas and information weighting methods to construct probabilistic models  in both The main function of this module is to process laser point cloud and image data including lidar–camera extrinsic calibration, ground plane removal of point cloud, region of interest (ROI) selection, obstacle clustering, and YOLO real-time target detection based on images and point cloud projects.s as the scaling factor, Before projecting the point cloud data onto the image frame at the corresponding moment to fuse the camera and laser information, it is necessary to calibrate the lidar–camera extrinsic parameters. The conversion of the point cloud to the pixel coordinate system is shown in roach in  to constd. The SVD decomposition of n points can obtain A large number of ground point clouds will greatly increase the computational cost of state estimation. To remove the ground better, the RANSAC algorithm  combinedExtensive experimental results indicate that when This paper combines the visual information to improve the consistency between the laser data and the visual observation information to crop the point cloud. For this reason, we only select the points that can be projected into the image. After performing the lidar–camera coordinate transformation according to Equation , the poiAs an efficient target detection model, YOLOv5 is equipped with the ability to perform real-time target detection on video stream data , and cann closest points with distance calculated as follows:r is a known distance threshold. Next, we select a point M and repeat the previous steps until no point is added to the set to end the search. The point cloud clustering effect is shown in After the above processes, the moving objects will be independent as point cloud clusters, so we use Euclidean clustering to cluster the point clouds based on the Euclidean distance between points, and cluster each frame into multiple groups to identify the moving obstacles better. We select a seed point In the literature , the poiRough processing stage is the first stage of the MoTI algorithm. The purpose of this stage is to quickly classify the point cloud roughly into static and dynamic points in order to improve the speed and accuracy of the nearest neighbor search in the second stage of precision processing. Therefore, we convert the point cloud into a range image and store it as a matrix for easy mathematical operations.Unlike the ray projection method and grid occupancy method ,30, thisT lie on the surface of the reference frame,k moments, respectively. The following conditions must be satisfied for the same static points in adjacent frames.m, as well as the variance Since the vehicle is also moving, we must align and match two adjacent points of the point cloud to determine whether the depth of the same point in space changes at adjacent moments. The point clouds of adjacent moments are iteratively aligned by normal distributions transformation (NDT) . NDT useWe set the current frame point cloud as Since the way is a statistical result, there is a statistical error, and some fixed points exist in the filtered dynamic points, so further selection is needed.The precise processing stage is the second stage of the MoTI algorithm. The objective of this stage is to search the rough processed point cloud for the nearest neighbor points and establish the movement evaluation probability model based on the search results and visual identification for each obstacle cluster. As the point cloud is filtered during the rough processing stage, the data dimension is further reduced, and the storage of the point cloud index is performed using map value pairs to enhance data processing efficiency. This ensures that the two frames of the point cloud after rough processing can avoid the data explosion problem after downsampling.To achieve this, we use K-nearest neighbor search and radius search, as shown in k, we iterate m is the serial number of the point cloud cluster after clustering, and The nearest neighbor search is performed on the point cloud sets For the accurate identification of dynamic points, we define the dynamic probability of the clustered point cloud as follows:projk=f alignments, which calculate the positional transformation and perform nonlinear optimization for edge points and planar points, respectively.In contrast, the mapping frequency is slow, and the map alignment is performed at 1 Hz. The mapping algorithm uses scan-to-map alignment to match the current frame point cloud with the submap, similar to Equations –23). Th. Th23). Therefore, after the previous multiple stages of point cloud processing and downsampling work, scan-to-scan and scan-to-map matching speed will be significantly improved, and after the moving targets in the point cloud are separated, the alignment accuracy is further improved.To evaluate the performance of the MoTI algorithm, we conducted dynamic point cloud detection and recognition, as well as map construction based on the KITTI dataset  and actual scenario. We also compared the localization performance with LOAM and LeGO-LOAM algorithms and performed a comparative analysis of the localization effect of the algorithm using the EVO tool. It should be noted that the LOAM algorithm cannot detect dynamic point clouds, and therefore, leaves noise blocks on the map during the mapping process; LeGO-LOAM is able to filter the point clouds on moving objects by extracting specific feature points from the segmented point cloud.The overall experiment consists of three parts. The first part illustrates the effectiveness of the two stages of the MOTI algorithm for dynamic point cloud processing with the KITTI dataset. The second part is an experiment on detecting dynamic points in real scenario. The third part is the experiment of localization and mapping based on the completed detection of moving objects.The calibration of the sensor external parameters, and the result are shown in Combining two experimental results from the KITTI dataset and the low dynamic dataset in natural scenes, the performance of the MoTI algorithm for detecting point clouds on moving objects is demonstrated.To better compare the performance of the algorithm, we used the KITTI dataset with ground truth for verification of the performance of the localization algorithm. Meanwhile, we conducted similar tests in real scenarios and analyzed the performance of the MoTI algorithm for localization. Hence, we conducted the following experiments.® CoreTM i7-8565U CPU and NVIDIA GeForce MX250 for testing and experiments.Outlier points have been accurately detected based on previous processes. The filtered point cloud is used as input for LOAM to improve localization accuracy and mapping effects. This study was conducted on the Lenovo ThinkPad T490 hardware platform, equipped with an IntelThis paper investigates the interference of moving targets in the point cloud with the SLAM algorithm. Pre-processing operations are first performed, including ground point cloud removal, obstacle point cloud clustering, point cloud-to-image projection, and YOLO detection. The pre-processed point cloud and target detection information are used as input to the MoTI algorithm, and a probabilistic model is constructed by a method of information weighting and statistics during the two stages of rough and precise processing to enable accurate detection of moving targets. Through experiments with different datasets, the effectiveness of the MoTI algorithm for the detection of moving objects in point clouds is verified, and it is also demonstrated that the MoTI algorithm acting on the front-end of LOAM can improve the positioning accuracy and mapping effect of the system, which improve approximately 12.8% of the localization accuracy compared to LOAM. Multiple comparative experiments conducted on the KITTI dataset and the actual scenario dataset demonstrate the effectiveness of the proposed algorithm.However, when the robot is relatively stationary with moving objects, it can be difficult to accurately identify them due to the insignificant change in depth difference. During the dataset processing, we observed that relatively stationary moving objects still retain some residual images in the point cloud map. In further planning, we will add further spatiotemporal correlation or loop closing to enhance the robustness of the system.


Recently, specifically designed video codecs have been preferred due to the expansion of video data in Internet of Things (IoT) devices. Context Adaptive Binary Arithmetic Coding (CABAC) is the entropy coding module widely used in recent video coding standards such as HEVC/H.265 and VVC/H.266. CABAC is a well known throughput bottleneck due to its strong data dependencies. Because the required context model of the current bin often depends on the results of the previous bin, the context model cannot be prefetched early enough and then results in pipeline stalls. To solve this problem, we propose a prediction-based context model prefetching strategy, effectively eliminating the clock consumption of the contextual model for accessing data in memory. Moreover, we offer multi-result context model update (MCMU) to reduce the critical path delay of context model updates in multi-bin/clock architecture. Furthermore, we apply pre-range update and pre-renormalize techniques to reduce the multiplex BAE’s route delay due to the incomplete reliance on the encoding process. Moreover, to further speed up the processing, we propose to process four regular and several bypass bins in parallel with a variable bypass bin incorporation (VBBI) technique. Finally, a quad-loop cache is developed to improve the compatibility of data interactions between the entropy encoder and other video encoder modules. As a result, the pipeline architecture based on the context model prefetching strategy can remove up to 45.66% of the coding time due to stalls of the regular bin, and the parallel architecture can also save 29.25% of the coding time due to model update on average under the condition that the Quantization Parameter (QP) is equal to 22. At the same time, the throughput of our proposed parallel architecture can reach 2191 Mbin/s, which is sufficient to meet the requirements of 8 K Ultra High Definition Television (UHDTV). Additionally, the hardware efficiency (Mbins/s per k gates) of the proposed architecture is higher than that of existing advanced pipeline and parallel architectures. The creation of intelligent sensor nodes that enable intelligent processing for Internet of Things (IoT) surveillance, remote sensing, and smart city applications is gaining more and more attention . In thisThe HEVC standard’s coding structure primarily comprises Prediction, Estimation, Motion compensation, Quantization and Transform, and Entropy coding. The video pixel value, which is broken down into two chrominance channels and one brightness channel, serves as the input for this coding system. The image is chunked into coding tree units (CTUs), which support a range of sizes . Intra aEntropy coding is a coding class that performs lossless coding based on the information entropy principle. Contrary to the Context-based Adaptive Variable Length Coding (CAVLC) and CABAC hybrid coding approach employed by the previous generation video coding standard AVC/H.264 , HEVC/H.A high-performance entropy encoder remains one of the hardware implementations’ constraints for entropy coding in video coding. The amount of data that must be processed via entropy coding in HEVC is also significantly increased to handle more complex Rate Distortion Optimization (RDO) operations and Syntax Elements (SEs), which places more demands on hardware implementation. Parallel processing is challenging to implement because of the stringent data reliance of the binary arithmetic coding employed in CABAC, as well as the complexity of the arithmetic coding procedures, which might make it challenging to increase the primary frequency ,13. DingThe throughput rates of the SE generation and processing module and the BAE module are essential because they are two modules that both supply and process data. Consequently, we must address the latency that the complex data preparation required by the higher-level modules results in. Wahiba et al.  propose There is a problem that they need to address specifically, even though the current work considerably increases the throughput of CABAC encoders. When encoding successive bins of the same context model in BAE, the pipeline or parallel architecture of CABAC periodically stall, decreasing the coding efficiency. This paper aims to improve the performance further and enhance the compatibility of the entropy coding module, which is used to ensure the overall video coding architecture and the continuous and stable operation of this entropy coding encoder. This study builds on our earlier work by offering several fresh architectural modifications to enhance the critical path delay and the number of bins provided every clock cycle, dramatically increasing the overall throughput. Below is a summary of this paper’s significant contributions.We examine the challenges and bottlenecks in pipelined or parallel implementations brought on by arithmetic coding’s back-and-forth dependency on coding states. We propose to use pre-range update and pre-renormalize technique to reduce the multiplex BAE route delay due to the incomplete reliance of the encoding process.We propose the variable bypass bin incorporation (VBBI) technique, which allows an extra two bypass coding bins to be processed in the same clock cycle in a quad parallel architecture, significantly improving the throughput of BAE in a parallel architecture.When the context model cannot be prefetched early enough, the pipeline will stall since the context model needed for the current bin typically depends on the outcome of the previous bin. We provide a prediction-based context model prefetching strategy to address this issue. Additionally, the Multi-result Context Model Update (MCMU) architecture is proposed, the critical path for state transitions is shortened by the context model update of the meticulously optimized parallel architecture.Based on the HEVC video coding standard, a highly compatible hardware architecture for entropy encoding is provided. The whole entropy encoding architecture is pipelined, and the data interaction between binarization and BAE is cached using parallel-in-parallel-out (PIPO) to improve the stability of the entropy encoder. It also develops a quad-loop cache architecture to improve compatibility for data interaction between the entropy encoder and other video encoder modules.As depicted in The binary symbol bin is the data that can be processed directly by the arithmetic coding module. Arithmetic coding is primarily split into Regular Coding and Bypass Coding, with various SEs accessing distinct selection criteria for each. Among them, the context modeling part will supply the context probability model of the associated bin based on the context data from the SEs for the regular coding bin.The HEVC standard defines several binarization methods for entropy coding: Fix-Length (FL) coding, Truncated Rice (TR) coding, K-order exponential Golomb coding, etc. The above binarization methods are the most critical for syntax elements in HEVC, except for very few syntax elements with their own specific binarization methods. This is mainly influenced by the numerical characteristics of different SE values and is related to the context model selection methods corresponding to other SEs. In addition, although the binarization method of SEs is specified directly by the standard, the quantization parameters cMax and RiceParam often depend on the specific encoding situation. For example, the cMax parameter of the merge_idx index is determined by the number of merge mode candidates.The probability of encoded blocks and encoded SEs is reflected in the context model in entropy coding. The core of context modeling is to estimate the distribution probability of the currently encoded SEs and enhance coding efficiency by using video data’s spatial and temporal correlation. The accuracy of context modeling, which holds a key place in the entropy coding standard, significantly affects the coding effect. For the standard coding model, the coding procedure for each bin includes the corresponding context model. To adaptively make adjustments to diverse videos, these context model need to be updated in real time.Although the arithmetic coding specified by the HEVC standard is conceptually comparable to the joint finite-precision binary arithmetic coding, numerous modifications have been made to the implementation techniques to reduce the complexity of the operations. The More Probable Symbol (MPS) and Less Probable Symbol (LPS) definitions of the encoding’s binary symbols denote the symbols having a big and small probability of occurrence, respectively. The binary arithmetic encoding inputs are the bin to be encoded and its accompanying contextual model. The pipeline architecture is one successful approach to increasing the throughput of BAE hardware, and the multi-channel parallel architecture is another. BAE in HEVC suffers from a huge area of memory due to lots of context models ,25, so SFor multiplexed parallel context model update architectures, the resulting path delay corresponds to many levels of multiplexers, which will dominate the critical path of CABAC. Thus, bottlenecks have emerged in determining how to improve the efficiency of the pipeline/parallel structure, and use less hardware to achieve better throughput CABAC designs.For a variety of data, other video encoder modules communicate with the entropy encoding. Numerous data will be combined in the entropy coding. The entropy coding may occasionally fail to finish digesting the input data in a timely manner, resulting in the loss of the input data since its coding efficiency differs from that of the other modules of the encoder. The residual coefficient data are the largest class of data among the coded data required for entropy coding. It also becomes challenging to balance the data supply of the reconstruction module with the value of the entropy coding and how to store these data more effectively.One of the features of CABAC is that each time a regular encoding is performed, the probabilistic model of the current encoded bin needs to be updated. The context modeling needs to transmit the same throughput to support the BAE with multiple bins constructed above.However, pipeline or parallel implementation is complex when faced with some exceptional cases. When there are successive bins with the same context model, since one clock is required to update the context model and both reading and writing of RAM data also occupy one clock, the latter bin cannot be read from the context model memory CM_RAM in the adjacent clock cycle that has not yet been written to the updated context model. To cope with the phenomenon of pipeline stall, this paper proposes a context model prefetching strategy and optimizes it for the multi-bin case, aiming to achieve a stall-free pipeline and low resource and high master frequency.The context modeling architecture of the pipeline BAE in this paper is shown in When the index of the current clock cycle input to CM_RAM is the same as the previous clock cycle, the context model is directly communicated to stage 1 of the pipeline through the Same_flag, and the context model is directly passed through the internal pipeline. The CM update module receives the updated model in the previous clock cycle instead of using CM RAM. Thus, regardless of whether the context model of the next bin is the same as the current consistent one, the correct data can be output promptly. The context model that needs to be updated is found in the state transition table and then saved in CM_RAM for real-time updates. This is a prefetching strategy proposed in this paper to solve this case, implemented by caching the model’s index.As depicted in Ref.  had to mSuppose there are bins with the same context model in the next clock cycle. In that case, the context model needs to transfer to the BAE module first through the cache in time to avoid the untimely transmission of the context model due to the read and write time of the memory.range, calculation of low, and the renormalization procedure are all carried out in a cascading manner, as is already noted, making the entire coding process feasible for pipelining activities.The context model update, computation of low of the current coding interval depends on range unidirectionally. In contrast, range does not depend on low. Therefore, in the design of the arithmetic coding pipeline, range and low can be calculated separately, and range is calculated first before low to shorten the critical path of the pipeline.A valuable fact for designing a single-way arithmetic coding pipeline is that the range computation includes rLPS lookup, interval recursion, renormalization lookup and shift; range update depends on rLPS, and rLPS depends on the current range and context model state. Therefore, compared with the low update and bitstream output part, the computation of range is the most complex part of the pipeline, and the loop algorithm generated by renormalization becomes a bottleneck for the hardware architecture.According to range, which makes it difficult to pipeline and affects the encoding efficiency. Since only the shifting of range and low and the counting of bitOutstanding are performed in the renormalization process, multiple renormalizations can be completed in one operation. The times of renormalizations differ when the bin is MPS and LPS. When the bin is MPS, if range is less than 256, then renormalization is performed once; otherwise, renormalization is skipped. Renormalization is required when the bin is LPS, as illustrated in However, one renormalization can only double the encoding interval of less than 256, so it is often necessary to perform multiple renormalizations due to the small range update is split into two levels of pipeline. Renorm_count is obtained from rLPS by indexing the table. The renormTab table size is 1 × 32, so in the context of the first stage pipeline to obtain four candidate rLPSs, we can also look up the table to obtain the renormalization number renorm_count which corresponds to the candidate rLPS, and shift to obtain the corresponding renormalization interval rLPS_renorm with four candidate values. So the renormalization count lookup table of rLPS can also be split into sub-operations carried out in the first stage pipeline. In contrast, the shift operation in renormalization is completely placed in the first stage pipeline.Therefore, this paper proposes pre-range update and pre-renormalize, as shown in the purple area of range [7:6] at the beginning of the second stage pipeline to determine the final rLPS renormalization interval. For the renormalization of MPS, the above pre-normalization method cannot be used because it depends entirely on the coding interval range of the previous encoding. In the second stage of the pipeline, the highest bit of rMPS is used as the judgment condition to determine whether to perform rMPS renormalization. The final range is selected between rLPS_renorm and rMPS_renorm according to whether the bin is MPS or not.The four candidate renormalization values obtained after pre-renormalize will also be used as indexes by low can be performed. If more multiple bypass bins are attempted, more cases need to be processed, which will be an extremely complex process that will consume a large hardware area and may become a new critical path. So this paper proposes area-efficient BAE pipeline architecture with compatibility.In the entropy coding of HEVC, the bins of regular and bypass coding are sequentially arranged. If hardware is designed separately for both, it can achieve very high coding speed under certain circumstances, especially for bypass bins. The bypass bin splitting (BPBS) described by , which irange remains unchanged, with only low changes. Our work integrates the bypass and regular encoding in one hardware architecture. The update of range is currently the critical path, so combining the update of bypass coding low into the pipeline does not cause the frequency to decrease.The coding state is calculated differently for different coding modes, as shown in low. The number of renormalizations when the bin is LPS is obtained via renormTab and LUT2 jointly checking the table. If the bin is MPS and rMPS is greater than or equal to 256, the renormalization is skipped, and if the bin is MPS but rMPS is less than 256, the renormalization is carried out once. The bypass coding only updates low, and its coding process is shown in the red part in The third stage of the pipeline architecture designed in this paper is the update calculation of A pipeline or parallel architecture are two efficient ways to increase the throughput rate of arithmetic coding technology. However, the dependency on the arithmetic encoding states makes the issue of long-timing routes in the pipeline structure even worse. This work presents a pipeline architecture for the arithmetic encoder and a multi-path parallel architecture with a single pipelined arithmetic encoder on each lane.range and low computed by the first encoder are used as the state input for the second channel, and so on for multiple channels of state updates. In particular, the encoding state of the last encoder will be saved in a register as the starting state data for the next set of four-way bin encoding.In the four-way parallel structure shown in low. In a basic four-bin BAE, either a regular or a bypass bin must be encoded sequentially. As we can see through the previous section, the update phase of range in a single-path pipeline architecture becomes the critical path, while the update of low is more straightforward.In the first stage of the pipeline, in addition to the pre-rLPScalculation and pre-renormalize, a pre-lookup table and storage structure for the renormalization count are added. The renormalization count candidates are stored in registers and will determine the final value in the second-stage pipeline, which will participate in the renormalization calculation of range, each time four bins are encoded, if immediately followed by one or two bypass bins, these two bins are added to the current bin sequence to achieve the maximum throughput rate of six bins encoded in at most one clock cycle. Even if the update of low increases to six bins at the same time, the critical path does not exceed the update process of the range of four bins, so there is no impairment in the main frequency performance, and the throughput of the parallel architecture can be effectively increased. Compared with the bypass bin separation architecture used in [So this paper proposes the Variable Bypass Bin Incorporation (VBBI) architecture, as shown in  used in , every sEntropy coding of HEVC is a module that performs statistically based lossless data compression of the results generated by other modules, so it is related to each module in video coding. The coding framework of the entropy coding module is shown in The data in the entire CTU generated in the video encoding process are passed to the entropy coding module; these data are diverse and need to be considered for hardware architecture to match the timing of transmission. However, the data processing speed of other modules and this module’s throughput will differ. Under the condition that the whole video coding is pipelined architecture, the data input structure, as shown in The input data include Depth_RAM , Intra_PU_RAM (luminance and chrominance direction), Inter_PU_RAM (information related to merge and amvp), Neighbor_RAM (information related to the top side and left side CTU), Residues_RAM , etc. The data to be entropy coded are cyclically cached through four RAMs. Each group of RAMs keeps all the data of one CTU, effectively reducing the dependency between video coding modules.The binarization schemes used for most of the SEs in HEVC are Truncated Unary (TU), Truncated Rice (TR), Kth-order ExpGolomb (EGK), and Fixed-Length (FL) codes. The rest of the SEs use their corresponding custom binarization schemes, which will include some compound encoding .Since the binarization is carried out separately for each SE and is not the bottleneck of the whole architecture, as long as the average throughput of the part is higher than the average throughput of the BAE, in any case, the entire architecture can be satisfied with smooth and efficient operation.The architecture of the single-core binarization module is shown in This design uses a parallel three-stage binarization scheme to meet the goal of smooth and efficient binarization, as shown in Experiments are conducted to evaluate the performance of the proposed architecture, and the superiority of the proposed CABAC encoder is tested via the HEVC reference software HM-16.7. The proposed CABAC encoder is implemented in Verilog HDL. RTL simulation is performed on 18 sequences in 5 classes. Tests cover All Intra (AI), Low Delay (LD), Low Delay P, and Random Access (RA) configurations and include settings for Quantization Parameters (QPs) 22 and 37.The CABAC pipeline 1 bin/clock architecture designed in this paper avoids the pipeline stall problem. This paper’s architecture follows the anticipated strategy, allowing it to avoid pipeline standstill brought on by context model updates and allocate the number of codes per group of bins through the prefetching module in the parallel architecture, which significantly increases coding efficiency. Our proposed CABAC encoder is implemented in Verilog. RTL simulations are performed using the bin sequences in For the full pipelined architecture CABAC, the gate count is 39.52 K, the maximum operating frequency is 714 MHz, and the maximum throughput is 714 Mbin/s. For the CABAC with the highly compatible parallel architecture in this paper, the overall CABAC throughput, at 513 MHZ, is 2191 Mbin/s. Numerous predictive lookup tables and alternative algorithms are required to raise the frequency and the number of parallel bins, and these efforts have led to higher throughput. Therefore, the throughput rate is also the highest due to the optimization of the hardware design and more advanced processes in this work.range update, context model accesses using RAM only, binarization using a single core and more advanced ASIC processes. Ref. [es. Ref.  parallelThe occurrence of consecutive bins in the same context model can cause stalls in the hardware pipeline architecture. To overcome this problem, we propose a prediction-based context model prefetching strategy to alleviate data dependencies by predicting the next bin model, and reduce critical path delays through the MCMU. In addition, we use pre-range update and pre-renormalize technique to reduce the multiplex BAE’s route delay due to the incomplete reliance of the encoding process. Then, we propose the VBBI technique to improve the throughput of BAE in a parallel architecture. Moreover, the data interaction between CABAC modules is optimized. In accordance with the experiments, our architecture eliminates pipeline stalls and saves encoding time, and works better for high resolution and low QP values, which is in line with the need for more high-definition videos as time progresses. Moreover, the throughput is enhanced and the hardware efficiency of the pipeline architecture is maximized. In future study, we will focus on making this work compatible with the multi-channel parallel architecture and Versatile Video Coding (VVC/H.266) hardware design .


Because of their superior performance, flexible strain sensors are used in a wide range of applications, including medicine and health, human–computer interaction, and precision manufacturing. Flexible strain sensors outperform conventional silicon-based sensors in high-strain environments. However, most current studies report complex flexible sensor preparation processes, and research focuses on enhancing and improving one parameter or property of the sensors, ignoring the feasibility of flexible strain sensors for applications in various fields. Since the mechanical properties of flexible sensors can be well combined with rubber conveyor belts, in this work polydimethylsiloxane (PDMS) was used as a flexible substrate by a simple way of multiple drop coating. Graphene-based flexible strain sensor films that can be used for strain detection at the joints of steel cord core conveyor belts were successfully fabricated. The results of the tests show that the sensor has a high sensitivity and can achieve a fast response (response time: 43 ms). Furthermore, the sensor can still capture the conveyor belt strain after withstanding high pressure (1.2–1.4 MPa) and high temperature (150 °C) during the belt vulcanization process. This validates the feasibility of using flexible strain sensors in steel wire core conveyor belts and has some potential for detecting abnormal strains in steel wire core conveyor belt, broadening the application field of flexible sensors. With the advancement of technology, flexible strain sensors are preferred over silicon-based sensors due to their excellent flexibility and ductility ,2,3,4,5.In the current report, flexible sensors are widely used in many different fields, and extensive research has been conducted, particularly in the application of wearable technology and human information monitoring. In contrast, in this work, we propose a new application area for flexible sensors, namely strain detecting for steel wire core conveyor belts to identify abnormal strains in the rubber matrix as well as in the internal steel cord. Belt conveyors are a kind of mechanical equipment widely used in large-scale continuous transportation; the steel wire core conveyor belt is its main load-bearing component. The conveyor belt consists of internal steel cord and rubber matrix. The steel wire core conveyor belt in a conveyor is not a complete monolith but is joined together by joints. These joints are the most vulnerable parts of the belt. When subjected to external loads, the bonds between the steel cords and the rubber in the joints of a belt are more prone to accumulate damage. When large deformations occur in the joint area, some of the cords may loosen and pull away from the rubber matrix. When the joint is loosened and then loaded again, the damage increases, and in severe cases, the belt tears ,28. A nuThis study investigated the viability of flexible strain sensors used in steel wire core conveyor belts, to broaden their application in engineering fields, and to monitor the working efficiency and dependability of the sensors after being exposed to high pressure and high temperature. Using a quick preparation procedure, we were able to successfully fabricate a flexible graphene-based strain sensor. After performance and sensitivity tests, it was applied to conveyor belt joints via embedding the sensor film into the belt. During manufacturing steel wire core conveyor belts, the process of vulcanizing experienced high pressure and high temperature. The graphene-based flexible strain sensor is flexible, malleable and conformal. In addition, the softness of the rubber layer of the conveyor belt provides a good cushioning effect for the sensor inside. Therefore, it is possible to apply flexible strain sensors to the interior of steel wire core conveyor belts. Detecting abnormal strains in conveyor belts by flexible sensors will help to improve the reliability of steel wire core conveyor belt joints. At the same time, it is possible to understand the performance of flexible strain sensors after withstanding higher pressure and high temperature, and the results provide some support and reference for the application of flexible strain sensors in other engineering fields.The fabrication process of the graphene-based flexible strain sensor and the obtained sensor morphology are described in detail in the In this experiment, monolayer graphene powder used as the modifying material for the flexible strain sensors was provided by Nanjing XFNANO Material Science and Technology Co., Ltd. . The diameter of graphene powder was 0.5–5 μm, the maximum thickness was 0.8 nm, and the monolayer rate was 80%. Polydimethylsiloxane (PDMS) was used as the matrix material for the flexible strain sensors. Anhydrous ethanol was used as the dispersant of graphene powder. The mass ratio of PDMS  to curing agent  was 10:1. They were mixed and stirred for 5–10 min before usage. The main materials required in the production of steel wire core conveyor belts are surface rubber , core rubber , and vulcanizing agent .− R0), R0 is the initial resistance before strain, and P is the pressure applied to the sensor. We applied pressure to the flexible strain sensor film using a circular indenter in the test bench, and −1 over the large response range. In addition, the flexible sensors exhibited a wide measurement range. Given that among the three sets of flexible sensors with different graphene fractions, the one with a graphene fraction of 1.5 mg/mL had the largest pressure range and the highest sensitivity to produce a large response, we chose to use this graphene fraction of the flexible strain sensor film for the subsequent testing and application in wire rope core conveyor belts. The sensor’s characteristics are shown in Before the flexible strain sensor film was vulcanized together with the wire rope core conveyor belt, we experimentally evaluated the sensitivity and reliability of three sets of flexible strain sensor samples with graphene component contents of 0.5 mg/mL, 1 mg/mL, and 1.5 mg/mL, respectively. The sensitivity performance S of the flexible sensor is evaluated as the amount of resistance change under a certain pressure, which is calculated as follows:0 of the sensor also exhibits periodic changes when the wrist and index finger joints are bent at periodic angles of 30° and 90°. The degree of resistance change of the sensor also increases with increasing bending angles. This can be explained by the fact that during bending, the sensor is stretched and its length relatively increases. As a result, the resistance change ∆R/R0 has a greater magnitude. The sensor’s deformation and the resistance change ∆R/R0 both decrease as the wrist and finger joints return to their initial states. This result shows that the flexible sensor can exhibit good sensitivity and responsiveness when subjected to bending. And our proposed flexible sensor has some potential for human body information detection. In addition, considering that the wire-core conveyor belt needs to run through the roller in practice, it is required that the flexible strain sensors have a good bending response if they are applied inside the conveyor belt. This also means that the sensor must be able to maintain stable performance over many bends and torsions. Therefore, ensuring that the flexible sensor has a good bending response is critical for the application of the sensor in wire rope conveyor belts.In addition to stress tests, we examined the characteristics of flexible sensor films’ performance under bending. To see the resistance change and response during bending, we placed the sensors on the wrist and the joint of the index finger. The resistance change characteristics of the sensor at the wrist joint and index finger are shown in In the current report, flexible strain sensors have been widely used in various fields, and most of the research work focuses on the application of wearable flexible strain sensors and human information detection. In this work, we envision the application of the proposed flexible strain sensors in wire rope core conveyor belts. The excellent properties of our proposed and fabricated graphene-based flexible strain sensor films, such as high sensitivity and faster response, were verified in previous tests. In addition, we consider the excellent conformality of the sensor and its ability to withstand large strains. Therefore, it becomes possible to apply it to the inside of steel cord core rubber conveyor belts. In a steel cord conveyor belt, the wire rope joints of the belt are the most fragile part of the belt, and when the belt is subjected to strain and damage, the wire rope at the internal joints will be the first to twitch, so we place flexible strain sensors in the joints inside the steel cord conveyor belt to detect and perceive the amount of pressure exerted on the conveyor belt and the strain exerted on the wire rope joints inside the belt.Using the experimental test rig, we applied pressure to the surface of a vulcanized steel cord conveyor belt and simply stretched the steel cord inside the belt. In both cases, we evaluated whether the flexible sensors inside the belt could respond to the pressure applied to the belt and the slightest twitching of the wire rope inside the rubber, in order to verify whether the flexible sensor film could still work properly after being subjected to a large amount of pressure during the vulcanization process of the conveyor belt, as well as the feasibility of applying the flexible sensors inside the wire-core conveyor belts. −1 under a wide pressure response range of 101 kPa), good linear response, and good reliability and durability. These excellent properties make the application of the sensors on conveyor belts possible. Subsequently, we designed the flexible strain sensors to be applied to conveyor belts and arranged the sensors at suitable locations on the belts. Meanwhile, the feasibility of graphene-based flexible strain sensor film inside the wire rope conveyor belt was demonstrated by making a specimen of the wire rope core conveyor belt and experimentally verifying it. The experimental results show that the sensor is not damaged by the huge pressure generated by the vulcanization process when making the conveyor belt because the buffer of the rubber layer in the conveyor belt plays a certain role. After the vulcanization process, the flexible sensor inside the conveyor belt can still work normally, and when pressure is applied to the surface of the conveyor belt or the wire rope in the conveyor belt is stretched, the flexible sensor is able to capture the corresponding force signals, and the resistance change curve of the sensor is basically consistent with the curve of the conveyor belt under force. The application of the sensor in the conveyor belt also proves that it can still work normally after withstanding large pressure, and at the same time, it provides a new method for detecting abnormal strain at the joints of the conveyor belt, which broadens the application field of the flexible sensor.In summary, we successfully fabricated a graphene-based flexible strain sensor film using a simple preparation process of drop coating and applied it to a steel cord core rubber conveyor belt in order to detect the strain of the belt and the internal steel cord. Prior to practical application, the sensor was briefly tested and found to exhibit a series of excellent characteristics under pressure and bending, including high sensitivity (maximum sensitivity of −0.0062 kPa


Target identification is an essential part of the drug discovery and development process, and its efficacy plays a crucial role in the success of any given therapy. Although protein target identification research can be challenging, two main approaches can help researchers make significant discoveries: affinity-based pull-down and label-free methods. Affinity-based pull-down methods use small molecules conjugated with tags to selectively isolate target proteins, while label-free methods utilize small molecules in their natural state to identify targets. Target identification strategy selection is essential to the success of any drug discovery process and must be carefully considered when determining how to best pursue a specific project. This paper provides an overview of the current target identification approaches in drug discovery related to experimental biological assays, focusing primarily on affinity-based pull-down and label-free approaches, and discusses their main limitations and advantages. Target identification is a crucial stage in the discovery and development of new drugs since it enables researchers to understand the mode of action of enigmatic drugs . For thiAffinity purification is a common method for identifying the targets of small molecules. In this method, the tested small molecule is conjugated to an affinity tag such as biotin or immobilized on a resin such as agarose beads. This chemically modified structure is used as a probe molecule that is incubated with cells or cell lysates. After incubation, the bound proteins are purified using the affinity tag. The purified proteins can then be separated and identified using sodium dodecyl sulfate-polyacrylamide gel electrophoresis (SDSPAGE) and mass spectrometry , 13–15. Below we discuss recent advances in affinity-based small molecule target identification techniques.The on-bead affinity matrix approach is a method used to identify the target proteins of biologically active small molecules using an affinity matrix . In thisBiotin is a small molecule that is commonly used in affinity-based techniques due to its strong binding affinity to the proteins, avidin and streptavidin. By attaching a biotin tag to a small molecule and then using it to pull out the target protein Fig. B, researUsing biotin-tagging over other protein isolation techniques has many advantages. This includes its low cost and simple purification and isolation of the target proteins. However, the high affinity of the biotin-streptavidin interaction requires employing harsh conditions in order to break their interaction and release the bound proteins from the resin . One comIn the photoaffinity labelling (PAL) approach, a chemical probe covalently binds to its target upon exposure to light. In this method, the probe design involves selecting a photoreactive group  and an affinity tag , 90 FigC. In thiThe PAL approach has several advantages, including a high degree of specificity, allowing for the labelling of the small molecule of interest in a manner that eliminates false positives and improves the precision of the results . AdditioDue to the integral roles played by the nature of primary molecule and the linker used to develop the probe, both these factors challenge the utilization of affinity-based pull-down approach. For example, while adopting this technique, it is important to pay attention to the design and synthesis of the modified probes. This step may require frequent testing and evaluation of different probes at different attachment points toward building a structure-activity relationships (SAR) model in order to produce probes that are both effective and specific. Thus, by carefully analyzing the SAR of a given probe arrangement, researchers can identify the structural features that are most vital to its function and use this knowledge to optimize the probe’s design . SAR stuOn the other hand, the linker or spacer group between the photoaffinity label in a photoaffinity probe and the chemical linker in a biotin-tagged molecule can have a significant impact on its performance. The use of photoaffinity linkers is limited by the possibility of nonspecific binding and random labelling of adjacent proteins. This can occur as a result of the linker’s reactivity, which can cause it to interact with proteins other than the intended target(s) , 103. InThe identification component, also known as the reporter tag, is designed to detect the presence of the probe and determine where it is bound to the target. There are numerous forms of available identification tags, including fluorescent dyes, radioisotopes, and particular binding partners like biotin and avidin , 106. WiLabel-free approaches utilize the small molecules in their natural state without undergoing any chemical modifications to their structures thus retaining their native confirmation and functional properties of the small molecule. This method is often preferred by the researchers as it does not demand modification or labelling of the primary molecule. Although this approach avoids any potential problems associated with compound labelling, it has few limitations as this label-free molecule can bind to unintended proteins and result in identification of false positive targets , 108. FuDrug affinity responsive target stability (DARTS) is a technique developed on the basis that small molecules can bind to and stabilize their target proteins, thereby increasing their resistance to proteolysis . DARTS utilizes this property to identify the target protein by detecting the binding-induced increase in proteolysis resistance , 110. To2O2). The rates of protein oxidation are quantified subsequently using mass spectrometry [The denatured proteins are more susceptible to oxidation than their native counterparts. The Stability of Proteins from Rates of Oxidation (SPROX) method takes advantage of this property by measuring the rates of protein oxidation levels of methionine residues on the protein’s surface in the presence and absence of the small molecule and detecting any changes that may be caused by the small molecule binding to and stabilizing its target protein , 111. Intrometry . The SPRtrometry . SPROX itrometry , 113.It is worth noting that both the SPROX and DARTS approaches are applicable to cell lysates, not living biological systems. As a result, they can only be used to study proteins isolated from cells rather than proteins within cells. This can limit the approaches’ application to specific research questions , 114.Cellular thermal shift assay (CETSA) was developed according to the concept of ligand-induced thermodynamic stabilization of protein targets. Its increased stability upon ligand binding can be assessed by determining the thermal stability of the protein . DiffereMutagenesis is a promising genetic tool to identify drug targets, which involves the manipulation of the expression or function of genes or proteins by altering a specific sequence of DNA or amino acids and observing the resultant effect of this mutation on the drug’s response . The mesMutagenesis offers the advantage of revealing a drug’s direct and indirect interactions with its target and other cellular components. Additionally, it can facilitate the optimization of lead compounds by modifying their structure or activity. Moreover, a large diversity of genetic variants generated by mutagenesis can be screened not only to detect drug sensitivity but also to understand drug resistance. Mutagenesis can help uncover primary and secondary targets and pathways involved in the drug’s mode of action . On the Genetic screening is another unique, unbiased method for cellular drug target identification , 122. InProtein target identification research is an important part of the drug discovery process and requires a significant investment of time and resources. Through careful consideration of the main advantages and limitations of affinity-based pull-down, label-free, mutagenesis and genetic screening approaches, researchers are able to make well-informed decisions when selecting a target identification strategy for drug discovery. Choosing the most appropriate approach depends on the specific requirements of the research project.


A realistic and visible dynamic simulation platform can significantly facilitate research on underwater robots. This paper uses the Unreal Engine to generate a scene that resembles real ocean environments, before building a visual dynamic simulation platform in conjunction with the Air-Sim system. On this basis, the trajectory tracking of a biomimetic robotic fish is simulated and assessed. More specifically, we propose a particle swarm optimization algorithm-based control strategy to optimize the discrete linear quadratic regulator controller for the trajectory tracking problem, as well as tracking and controlling discrete trajectories with misaligned time series through introducing a dynamic time warping algorithm. Simulation analyses of the biomimetic robotic fish following a straight line, a circular curve without mutation, and a four-leaf clover curve with mutation are carried out. The obtained results verify the feasibility and effectiveness of the proposed control strategy. As a result of the rapid growth of digital twin technology, game creation engines, such as Unreal Engine and Unity 3D, became a prominent focus of research in the construction of robot simulation systems ,2,3. TheIn recent years, underwater robots were utilized in a variety of marine applications, including marine resource exploitation, detection of submerged pipelines, underwater rescue, and marine environment monitoring . Some biMany control approaches, such as central pattern generator, model predictive control, neural network, synovial membrane control, and adaptive robust control, were developed by domestic and international researchers for underwater robot trajectory tracking ,9,10. LiQ and R matrices; the Q weighting matrix is related to the trajectory deviation of the state variable, and the R weighting matrix is related to the control quantity and actuator saturation. Nonetheless, the key to creating LQR-optimized controllers for real-time applications is largely dependent on the efficient selection of Q and R weighting matrices, which necessarily involve trade-offs and are often tuned based on trial and error [Q and R weighting matrices in an effort to solve the drawbacks of the trial-and-error method [In addition, the linear quadratic regulator (LQR) was effectively applied to a variety of complicated systems, such as double inverted pendulums, fuel cell systems, vibration control systems, electric cars, and; it is also widely employed in robot trajectory tracking control ,18,19. Tnd error . In addind error . Evolutind error . Deng etr method . Gupta er method . Elumalar method . Zhou etr method . HoweverFirstly, a visual simulation platform for underwater robots that simulates the real ocean environment is established. At present, the simulation environment developed based on the Unreal Engine is mostly employed to simulate the sky and land. This paper builds an Unreal Engine-based simulation scene of the real ocean environment and offers a dynamic and visualized underwater robot simulation platform.Secondly, a discrete linear quadratic regulator (DLQR) controller is designed for the biomimetic robotic fish. As the target trajectory in the simulation experiment is composed of discrete points, the LQR is discretized. Using the LQR controller, the simulation and comparison experiments that involve tracking the trajectory of the biomimetic robotic fish are carried out in three states: straight line, no-angle mutation curve, and angle mutation curve.Q and R matrices of the DLQR controller, in order to reduce the workload of testing and the influence of human factors and the local optimum, a trajectory tracking control strategy based on particle swarm optimization (PSO)-DLQR is proposed. For the time series misalignment problem of the discrete system tracking trajectory, a dynamic time warping algorithm is introduced as the performance index of PSO, which improves the convergence of the algorithm.Thirdly, the DLQR controller is further optimized using the PSO and DTW methods. When selecting the weighted This paper aims to use Unreal Engine to build a dynamic visualization simulation platform for underwater robots that simulates the real ocean environment, which is extensively applied to the simulation research of biomimetic robotic fish. The main contributions of the paper are summarized as follows:The remainder of this paper is organized as follows: In this paper, a single-joint model of the biomimetic robotic fish propelled using the caudal fin is developed, as depicted in b, and the center of mass of the swinging rear part is located at point t. The mass center from the rigid front part to the swinging rear part is represented by the vector As illustrated in Next, the force analysis is carried out from the rigid front part to the swing rear part. The dynamic model of the biomimetic robotic fish was established using the Newton–Euler formula. The hydrodynamic forces at its swinging rear are generated via additional mass forces and viscous drag. The following equation describes viscous drag.The force and moment for the swinging rear can be expressed asThe forces and moments on the rigid front can be expressed as the sum of the forces from the swinging rear and the hydrodynamic force.The velocity and angular velocity of the swinging rear can be converted into the representation of the velocity and angular velocity of the rigid front.F is the force on the swing rear part, M is the moment on the swing rear part, C is the viscous coefficient, Through combining Equations (1)–(4), the kinematic parameters of the rigid front part of the biomimetic robotic fish can be expressed asThe path followed by the biomimetic robotic fish in this paper is in a horizontal plane with a fixed depth; thus, the motion error model of the biomimetic robotic fish in the world coordinate system only considers the two-dimensional case. Consequently, only the yaw angle The LQR controller is employed to solve a biomimetic fish’s trajectory tracking problem, which, in turn, solves a continuous system of state quantities to track a reference trajectory. However, in general, the target trajectory is a discrete trajectory composed of discrete points; thus, the LQR controller is discretized to obtain a DLQR controller. The state space equations of the discrete control system can be expressed asQ is a semi-positive definite matrix that constrains the state cost of the system, R is a positive definite matrix that constrains the input cost of the system, and the control law which yields the best response is shown as follows:The purpose of the DLQR controller is to make the biomimetic fish trajectory tracking state error as small as possible while maintaining the input of the control quantity as small as possible to ensure the control stability. Thus, the objective function is defined as follows:P is designed, and the optimal state feedback matrix K can be obtained as follows:P is the solution of the Riccati algebraic differential equation.In order to stabilize the control system, a constant matrix Q and R. Therefore, selecting a suitable weighting matrix parameter is the key to controlling the trajectory tracking. In the previous parameter selection process, the parameters of Q and R matrices are usually obtained via experimental or trial-and-error methods and summarized according to personal experience, which is often not the optimal solution. To achieve more accurate control of robot trajectory tracking, this paper introduces the Particle Swarm Optimization (PSO) algorithm to optimize the parameter selection of Q and R matrices, which improves the trajectory tracking control performance of traditional DLQR controllers to a certain extent.When using the DLQR controller to control the robot’s trajectory tracking, the tracking performance’s evaluation index is determined based on the weighting matrices In the design process, the desired trajectory and the actual trajectory calculated using the DLQR controller are trajectories composed of discrete points. The errors of the corresponding points are significant and inevitable because their time series are not aligned; thus, the objective optimization function of DLQR cannot be directly used as the fitness function of the PSO optimization algorithm. Since trajectory tracking mainly evaluates the overlapping similarity of two trajectories, a common approach is to use the Euclidean distance to calculate the error distance of two trajectories. However, a growing number of studies show that Euclidean distance has serious drawbacks in measuring the similarity of time series. These drawbacks mainly lie in the fact that if two sequences are only unaligned at data points, while the trajectories are very close, the calculated Euclidean distance error will still be very large, which will not accurately measure the similarity of the two sequences. The Dynamic Time Warping (DTW) algorithm can effectively solve the above problem via a very clever dynamic planning method to achieve the dynamic regularization of the time axis, meaning that all data points have the best alignment . The algD is constructed for the sequences X and Y. Based on the constructed distance matrix D, a path is found from the upper left corner to the lower right corner such that the sum of the element values through which the route passes is minimized. Next, according to monotonicity and constraint, the previous point of i-th and j-th points of the two sequences.The DTW algorithm is defined as follows. Given a target sequence The algorithm calculates the sum of the distances of errors between two trajectories to measure the similarity between two time series trajectories and use it as the fitness function of the PSO algorithm. Compared with using the controller’s objective function to calculate the fitness value of the optimization algorithm, the computational efficiency of PSO is improved to a certain extent, thus effectively improving the convergence.The position and velocity of the biomimetic fish are set as the state quantities Q and R matrices are set as unit matrices, and the desired acceleration u is calculated.The initial Q and R are optimized using the particle swarm optimization algorithm, and the optimization flow chart is shown in The weighting matrices The specific optimization steps of the particle swarm optimization algorithm are as follows:N = 10, the feasible solution dimension as D = 6, and the maximum number of iterations as 100. Moreover, the Q matrix parameters are restricted to R matrix parameters are limited to w = 0.7298, the cognitive learning factor is Step 1: Particle population initialization: we initialize the number of populations as Q and R matrices, is brought into the Riccati algebraic differential equation to calculate the optimal state feedback matrix K. The trajectory tracking program of the biomimetic machine fish is run to correctly calculate and record the coordinate data points of the actual motion trajectory. The reference and actual trajectory are brought into Equation (11) to calculate and evaluate the fitness function values iteratively, and the optimal historical position pbest and the optimal global position gbest of the particle population are obtained.Step 2: The particle population individual Q and R matrix parameters and end the program. Otherwise, we continue the execution.Step 3: If the terminal condition is satisfied, we output the global optimal Step 4: We update the velocity and position of the particle and turn to Step 2 to continue the execution. According to the optimal historical position and the optimal global position of the particle, the velocity and position are updated using Equations (13) and (14).i particle at time slot t + 1, and E is expressed as follows:The fitness function The optimized weighting matrix isThe simulation environments developed via traditional robot simulation platforms, such as Gazebo, Webots, and CoppeliaSim (formerly known as V-REP), have multiple problems, such as simpler design and lack of ability, which make it difficult to build large and complex scenarios under guaranteed simulation realism ,29,30. TStep 1: Build an ambient light source. Add “Directional Light”, “Sky Light”, and “Visual Effects” from “Light Sources” to the “Sky Atmosphere”, “Volume Clouds”, and “Exponential Height Fog” in the viewport area. In order to simulate effects such as ambient light effect and refraction and reflection of the water surface, materials and writing scripts should be complemented.Step 2: Build the undersea landscape. In the “Landscape Mode”, draw the undersea landscape, add surface materials and undersea obstacles, and add some underwater elements to enrich the scene, such as seaweed, rocks, shipwrecks, etc.Step 3: Add the ocean water body. Add the “Water” plug-in in “Selection Mode”, and adjust the land and seafloor shape, curve, etc.Step 4: Simulate ocean waves. According to the Gerstner Wave formula, the offset value and normal value are calculated to simulate the effect of sharp crests of water waves. The Gerstner Wave calculation formula can be expressed asThe specific steps required to establish a UE-based ocean simulation environment are as follows:Step 5: Add the biomimetic robotic fish model and associate the model with AirSim. The control script is written in Python to run and call the control interface of the AirSim platform for real-time dynamic control simulation implementation in the ocean scene edited using Unreal Engine.The establishment process and the effect of the established UE-based dynamic visualization simulation platform are shown in Based on the built simulation environment, the trajectory tracking control simulation of the biomimetic robotic fish is performed before and after optimizing the weighting matrix parameters of the DLQR controller to track a straight line, a circular curve without angular mutation, and a four-leaf clover curve with angular mutation trajectory, respectively. In the simulation diagram, the solid green line is the reference trajectory, and the solid red line is the tracking trajectory. In the trajectory diagram, the black line is the reference trajectory, the red line is the DLQR-controlled tracking trajectory, and the green line is the PSO-optimized controlled tracking trajectory.In the process of tracking the straight line trajectory of the biomimetic robotic fish, the sum of the distance error between the reference trajectory and the tracking trajectory before the optimization of the weighting matrix obtained using the dynamic time warping algorithm is y direction is smaller because there is no abrupt change in the angle of the circular curve. However, the tracking trajectory of the DLQR controller before the parameter optimization has a large lag; thus, it leads to a larger tracking error.According to the dynamic time warping algorithm, the sum of the distance error between the target trajectory and the tracking trajectory before the optimization of the weighting matrix is The sum of the distance errors between the reference trajectory and the tracking trajectory before the optimization of the weighting matrix obtained using the dynamic time warping algorithm is Q and R weighting matrix parameters can track the target trajectory more quickly when using the biomimetic robotic fish to track straight and curved trajectory without abrupt angle changes; however, the tracking lag is larger. When tracking the curved trajectory with abrupt angle changes, it is difficult to adjust the control attitude quickly to track the target trajectory under the continuous large angle changes due to the large tracking lag, while the tracking effect is poor. The DLQR controller with optimized Q and R weighting matrix parameters based on the PSO algorithm can track the target trajectory quickly and with more limited tracking errors when using the biomimetic robotic fish to track straight trajectory and curve trajectory without abrupt angle changes. When tracking the curve trajectory with abrupt angle changes, the trajectory deviates due to the hysteresis of the tracking; however, it can still adjust the control attitude quickly and track the target trajectory.Synthesizing the above trajectory tracking testing results, we found that the Unreal Engine-based ocean simulation environment can make the experimental process more realistic and intuitive, and can effectively observe the dynamic changes of the environment and the real response of the experimental object during the experiment, which can effectively improve the dynamic simulation experiment effect. The test results indicate that the DLQR controller with initial To further analyze the control effectiveness of the DLQR method, we compared it to the model predictive control method. At each sampling moment, the model predictive controller’s current control action is determined through solving a finite time domain open-loop optimal control problem. The current state of the process is viewed as the starting point for the optimal control problem, and the optimal control sequence found only implements the first control action. In Equation (17), we allow the robotic fish’s position and velocity to be state variables, and allow the acceleration to be control input variables.Therefore, the state equation of the system can be expressed asWhen If we assume that The biomimetic robotic fish is then controlled using the model predictive control approach, which is also optimized using PSO algorithm, to track the reference trajectories, which are a straight line, a circle, and a four-leaf clover-shaped curve. The DTW distance and average DTW of trajectory tracked through the optimized DLQR method and the optimized MPC method are shown in Q and R weighting matrices, the simulation results indicate that the DLQR controller’s accuracy can be effectively enhanced.This paper develops an Unreal Engine-based framework for dynamic simulation of real ocean environments. It integrates the AirSim simulation system to visualize the trajectory tracking control of the biomimetic robotic fish and observe the motion state of the biomimetic robotic fish in real time as it swims in the ocean environment, resulting in a more realistic simulation effect and a higher degree of visualization. On this dynamic simulation platform, we propose a control strategy based on the PSO algorithm to optimize the DLQR controller for the trajectory tracking problem of the biomimetic robotic fish, as well as to track and control discrete trajectories with misaligned time series through introducing the dynamic time warping algorithm. After employing PSO to optimize the Although this paper gives some fascinating simulated examples of trajectory tracking of biomimetic robotic fish under perfect settings, it does not take into account aquatic environmental influences, and the control model design is also relatively simplistic. To further investigate the dynamic control mechanism of the biomimetic robotic fish operated in ocean scenarios, we will increase the complexity of the control model and further explore the impact of environmental changes and obstacles in future work.