Realizing robust six degrees of freedom (6DOF) state estimation and high-performance simultaneous localization and mapping (SLAM) for perceptually degraded scenes  is a challenge in robotics. To solve these problems, we propose a SLAM algorithm based on tightly coupled LiDAR-IMU fusion, which consists of two parts: front end iterative Kalman filtering and back end pose graph optimization. Firstly, on the front end, an iterative Kalman filter is established to construct a tightly coupled LiDAR-Inertial Odometry (LIO). The state propagation process for the a priori position and attitude of a robot, which uses predictions and observations, increases the accuracy of the attitude and enhances the system robustness. Second, on the back end, we deploy a keyframe selection strategy to meet the real-time requirements of large-scale scenes. Moreover, loop detection and ground constraints are added to the tightly coupled framework, thereby further improving the overall accuracy of the 6DOF state estimation. Finally, the performance of the algorithm is verified using a public dataset and the dataset we collected. The experimental results show that for perceptually degraded scenes, compared with existing LiDAR-SLAM algorithms, our proposed algorithm grants the robot higher accuracy, real-time performance and robustness, effectively reducing the cumulative error of the system and ensuring the global consistency of the constructed maps. SLAM is the basis for autonomous navigation in robots , which uDespite the superior performance of 3D LiDAR, scenes characterized by single spatial structures, complex environments, uneven illumination , narrow Robust 6DOF state estimation and high-performance SLAM can be used to enable robots to more effectively deal with these special perceptually degraded scenes. In this paper, we propose a tightly coupled LiDAR-SLAM fusion algorithm with two sensors, LiDAR and IMU, as input information sources. The framework of the system consists of two parts: front end iterative Kalman filtering and back end pose graph optimization, these are less dependent on external features, have a strong ability to adapt to the environment and are especially suitable for perceptually degraded scenes such as corridors and roadways. However, in perceptually degraded scenes, ensuring the tradeoff between the estimation accuracy, computational complexity and robustness of SLAM systems is still a major problem faced by tightly coupled LIO . Therefo(1)We propose a tightly coupled LiDAR-IMU fusion SLAM algorithm that uses both LiDAR and IMU sensors as input information for perceptually degraded scenes. The results show that our algorithm improves the accuracy, real-time performance and robustness of robots working in such scenarios.(2)Our algorithm achieves robot state updates by establishing an error-state-based iterative Kalman filter at the front end and adds loop detection and ground constraints in the tightly coupled framework at the back end, which effectively reduces the cumulative error of the system and ensures the global consistency of the constructed maps by optimizing the relative poses between adjacent keyframes.(3)Our algorithm fuses different types of sensors in a tightly coupled framework to achieve robust 6DOF state estimation and high-performance SLAM for perceptually degraded scenes and outputs motion trajectories, a global map and state estimation in real time after global graph optimization.The main contributions of this paper are as follows:The remainder of this paper is organized as follows: In Of the many studies related to LiDAR-SLAM, we focus on work related to SLAM algorithms for 6DOF state estimation and multi-sensor fusion in robots. The different fusion frameworks of these algorithms are usually classified as loosely coupled and tightly coupled.LiDAR odometry is usually performed by using scan-matching methods such as ICP  and GICPThe loosely coupled fusion method usually processes the measurement of each sensor input separately and weights the measurement results to infer the motion state of the robot. LOAM  and LeGOUnlike loosely coupled methods, tightly coupled methods typically fuse raw LiDAR feature points (rather than the scanning registration results) with IMU data, which yields a high robustness and accuracy, and can be divided into optimization-based and filter-based methods.An optimization-based method  introducThe most significant difference between our approach and other tightly coupled methods is the specific environment of perceptual degradation. We have chosen a filter-based optimization approach at the front end to improve the system’s robustness, which often uses an extended Kalman Filter (EKF). However, the EKF can easily cause a system to be affected by linearization errors . With reOur system receives data from 3D LiDAR and IMU sensors. The goal of the system is for a robot to perform real-time 6DOF state estimation and build a globally consistent map. The system framework of our proposed algorithm, which is shown in z-axis of the world coordinate is defined as coinciding with the normal to the surface of the Earth’s ellipsoid, with the other two axes lying in the plane orthogonal to the z-axis, which conforms to the right-hand rule. To clearly describe the ranging process, we define three reference coordinates. Additionally, we assume that the IMU and robot coordinates are consistent, the origin of the world coordinates is the origin of the robot coordinates when the coordinate is initialized, and the robot coordinates are consistent with the center of the LiDAR and IMU coordinates. Because the connection between the sensors is rigid, the external calibration between the LiDAR and IMU sensors is obtained by offline calibration. The k and bk+1, can be expressed as:The IMU pre-integration algorithm was proposed by Geneva . To avoiDue to the rotational mechanism inside the LiDAR sensor, the LiDAR measurements are not synchronized and, therefore, the current frame point cloud is prone to distortion. The LiDAR moves at constant angular and linear velocities at a constant speed. Therefore, the timestamp information of each point can be obtained, a linear interpolation operation used to obtain the pose transformation matrix corresponding to each moment, and the motion compensation of the current frame point cloud can be achieved by using IMU measurements ,14.ith subframe of the current laser frame, and i ∈ {1, 2,…, N}. Let the pose transformation matrix between two consecutive LiDAR frames in the time interval ith subframe for the LiDAR attitude transformation in the time interval The current frame point cloud is divided into N subframes by By performing the current subframe transformation, the motion compensated point cloud for S be the set of consecutive points i returned by the LiDAR in the same scan. Then, the smoothness c of the local surface can be defined and evaluated as:The feature extraction and matching method in this paper is similar to the one introduced in LOAM : the smoc value and the maximum c point is selected as the edge point, that is, the point on the sharp edge in 3D space, which varies greatly in size compared to the surrounding points and has a higher curvature and a higher smoothness. The minimum c point is selected as the plane point, that is, the point on a smooth plane in 3D space, which varies slightly in size compared to the surrounding points and has a lower curvature and a lower smoothness. The LiDAR scanning features of the extracted edge line and plane at time I are expressed as i is formed, which can be expressed as The feature points can be divided into two main categories: plane points and edge points. Then, the points in the scan are sorted according to the z-axis displacements of keyframe nodes when the ground robot works in different spatial regions while reducing cumulative errors by using more constraint information to improve the pose estimation accuracy. The key to establishing ground constraints is to accurately extract the road planes; determining how to spend the shortest amount of time to complete the planar segmentation and choosing a robust estimation method with few iterations, a high resistance to noise and random sample consensus (RANSAC) are also essential processes [2 + b2 + c2 = 1, d > 0,  is the plane normal vector and d is the distance from the LiDAR sensor to the plane. These four parameters can determine a plane.Inspired by , adding rocesses . Accordit.Because the robot may travel over uneven ground, it is not possible to construct pose maps with fixed planes as nodes as a constraint. To ensure that the constructed constraints are consistent with the actual situation, a subgraph-based ground extraction method is proposed. Based on the constructed local point cloud map, the true planar parameters are extracted as follows: The calculation of the error between the attitude node and the ground plane node  can be eTightly coupled LIO uses IMU measurements and features extracted by LiDAR in two consecutive scans to estimate the relative positional transformation of the robot. To avoid large linearization errors caused by growing uncertainty due to IMU measurements , we consk to k + 1 under the IMU coordinate, expressed as:To facilitate the derivation of the iterative Kalman filter, we define the state According to the basic principle of the ESKF , once tht and t.t and t.The equation of motion over linearized continuous time is defined as:Therefore, the equation of motion linearized in continuous time by Equation (12) is discretized  to obtaiThe observation model is mainly obtained by matching the features extracted by the LiDAR to the point cloud and is defined as: ith characteristic point in the LiDAR coordinate k + 1 frames, and the observation equation can be described ask+1th frame to the kth frame. For edge points, the distance is between point Given the error term in In this paper, an iterative extended Kalman filter (IEKF) is used and the state update can be abstracted as an optimization problem. Because the error state is composed of the deviation of the prior state and the residual of the observation model, the error state update problem can be turned into a joint optimization problem of solving for the minimum of the deviation of the prior state and the minimum of the deviation of the observation model relative to the pose solution, which can be expressed as:jth iteration. Note that in each iteration, new matching edges and matching surfaces are found to further minimize the metric error, so Thus, the Kalman gain equation for the ESKF iterative observation can be expressed as Equation (21), and the mean value of the posterior state can be expressed as Equation (22):nth iteration, and the variance of the posterior state can be expressed as:When By iterative updates, the exact posterior state The positional information was calculated earlier under the robot-centered reference coordinate. Therefore, to obtain global coordinate information, state synthesis is needed, and the synthesis formula can be expressed as:LiDAR and IMU sensors are used, where the acceleration deviation To keep a limited number of poses for estimation, on the back end, we deploy a keyframe selection strategy. The selection of keyframes can greatly improve the computational efficiency and ensure that the algorithm can run in real time on larger environmental maps.k−1th keyframe be kT−1 and the pose of the kth keyframe be kT [When selecting keyframes, matching errors and redundant keyframes must be reduced to decrease the computational effort, while sparse keyframes increase the uncertainty of interframe observations and degrade the map quality. Usually, the first point cloud is used as the keyframe, and the keyframe selection criteria based on the real-time point cloud are based on the relative positional change between consecutive frames. Let the pose of the me be Tk . The relWhen any of the above three criteria exceed the set threshold, the current kth frame of the point cloud is selected as a keyframe. The other LiDAR frames between the two keyframes are discarded. Adding keyframes in this way achieves not only a balance between map density and memory consumption but also a uniform distribution in space, which is suitable for real-time nonlinear optimization.k + 1 frames are selected as sub-keyframes, continuous sub-keyframe optimization is performed using ICP and the relative change in pose between sub-keyframes is kth sub-keyframe to the world coordinates as k+1th sub-keyframe to the world coordinate can be obtained as k + 1 sub-keyframes are converted into the world coordinates, and the converted sub-keyframes are merged into a voxel mapping iM consists of two sub-voxel maps, that is, the edge feature voxel map set When a point cloud frame is detected as a keyframe, the environment map of the current frame needs to be updated, and the main method is to convert the real-time point cloud information of the keyframe to the world coordinates by coordinate transformation . When k To reduce the cumulative error of the sensor, we add loop detection. When the robot detects all historical keyframes during operation, pose graph optimization is combined with all relevant positions and poses in the historical keyframes. When performing loop detection, the current real-time keyframe is first compared with the historical keyframe, and then the candidate loop frame is selected. The following conditions are met: (1) the index of the current keyframe is larger than the index of the history keyframe; (2) the difference between the track distance of the current keyframe and the history keyframe is larger than the set threshold; and (3) the relative translation distance between the current keyframe and the history keyframe is smaller than the set threshold. Finally, the candidate loop frame with the highest score is selected by comparing the candidate loop frames and the currently registered keyframes. The keyframes and candidate loop frames are added as nodes to the SLAM pose graph optimization, and the edge constraints are the relative poses obtained from point cloud registration.mP in the B coordinates and its relative pose change to the world coordinates is mT. If the keyframe database contains the keyframe pose set TD and the point cloud set PD, then a kd tree search with radius r is performed. Therefore, the problem to be solved is to show that the loop has been detected if we can find TD such that the sum of the root mean square error of the point cloud mP of the current frame after the relative change P is minimized. The keyframe pose TD registers the keyframe point cloud DP to the local point cloud map M constructed in the world coordinate space.Algorithm 1 Loop Detection Algorithm.Input: Output: 1: if  by solving the following cost function:Only the current pose of the point cloud in the world coordinates for the current frame is obtained, and To effectively solve the large-scale simultaneous localization and map building problem, the method of cluster adjustment (BA) is often used; that is, the map optimization uses sensor poses and spatial points. However, as the trajectory of the robot increases, the BA method decreases the computational efficiency of the system as the size of the map continues to increase. To address this issue, the pose graph optimization only uses the trajectory as a constraint for the pose estimation and no longer optimizes the pose estimation of the feature points. Based on bitmap optimization theory, the keyframe pose, ground constraint and loop detection constraint are calculated to construct a multi-constraint-based cost function for bitmap optimization.The pose graph optimization problem can be efficiently solved by standard optimization methods such as the Gauss–Newton or Levenberg–Marquardt (LM) algorithms . This prWe conducted a series of quantitative and qualitative analysis experiments on the performance of the proposed tightly coupled LiDAR-IMU fusion SLAM algorithm and compared the results with those of other state-of-the-art LiDAR-SLAM methods. All methods were tested under the same conditions. The hardware platform was an inspection robot with sensors and an on-board computer, as shown in Since accurate true values are not available for comparison for perceptually degraded scenes, in order to validate the performance of our proposed method  while ensuring the experimental soundness, we first used public dataset 1 and public dataset 2 for quantitative analysis. Note that all algorithms were tested without the loop module. To verify whether our proposed method causes the accuracy of the SLAM system to decrease as the speed of motion changes, we used the public dataset 1, which is a six-set fast-to-slow dataset comparing rotation and translation errors, whose true values are provided by the indoor high-precision motion capture instrument. To verify whether our proposed method increases the cumulative error of the system as it operates for a long time, we used public dataset 2, which is the 2011_09_30_drive_0028 sequence of the KITTI datasets, whose true values are provided by outdoor high-precision GPS. Subsequently, to verify whether our proposed method is robust under three different real perceptual degradation environments, we recorded three self-picked datasets with our own experimental platform, and the self-picked datasets were qualitatively analyzed for real promenade environments under three perceptual degradation scenes. Furthermore, we compared the method with the a_loam, lego_loam and lio_sam methods. We verified the excellent robustness of our proposed algorithm through horizontal comparison between different algorithms and vertical comparison between different scenes. In addition, we designed ablation experiments to verify the effects of adding ground constraints and loop detection constraints on the overall optimization results. Finally, a runtime analysis was performed for the main modules.The results show that our proposed algorithm always provided more accurate translational and rotational states for both fast and slow motions. The table also shows that, with motion compensation and online external reference estimation, LIO provided a better performance, especially in the case of fast motion, probably because more IMU excitation had to be generated. The proposed method, in contrast, was not affected by the speed of motion and performed better at any speed of motion relative to LIO.To evaluate how the error varied over time, we used the 2011_09_30_drive_0028 sequence from the KITTI datasets, with the four different algorithms producing roughly the same motion trajectory but with subtle differences in some areas. To explore the differences between the algorithms, the EVO tool was used for quantitative analysis. The final motion trajectories of the different methods are shown in To further refine the comparison experiments, we analyzed the maximum deviation, mean deviation, minimum deviation, root mean square error and standard deviation of the KITTI datasets 2011_09_30_drive_0028 sequence for the four different algorithms for each relative translation and rotation, as shown in The statistical results show that the proposed algorithm had smaller translation and rotation errors than the other three algorithms when tested on the same datasets. Therefore, the accuracy of the state estimation of our proposed algorithm is better than that of the other methods.We conducted three experiments with the same sensor configuration in different test environments to verify the performance of the proposed algorithm for perceptually degraded scenes. The datasets included corridor scene 1 on the 3rd floor of the mechanical building, corridor scene 2 on the 21st floor of the innovation building and corridor scene 3 on the 11th floor of the science and technology building. The proposed algorithm was analyzed and compared with the aloam, lego_loam and lio_sam methods, and its point cloud map was constructed in the current experimental environment, as shown in Z-axis. In the c scene, the structural symmetry in the annular scene was too similar, resulting in interference of the extracted features of the original point cloud data by a large number of similar point cloud features, and the entire SLAM system failed. Lio_sam constructed the geometric structure of the surrounding environment in a short period of time for scenarios a, b and c. However, as the mileage of the laser odometer increased, when the system detected the loop, the point cloud map construction failed because the scenarios were highly similar, resulting in the establishment of an incorrect loop detection constraint.As shown in In conclusion, the four different algorithms build different point cloud maps in the same environment because of their strengths and weaknesses. The scheme of aloam adopts the assumption of uniform motion model and uses IMU data for distortion correction of point clouds. The system has good real-time performance, but there is no loop detection module, which causes it to drift in large-scale tests. The lego_loam method is an optimized version of loam which segments and clusters the raw data frames scanned by LiDAR, labels the ground point cloud, and improves the accuracy of state estimation by two-step LM optimization. However, because it relies too much on the extraction of point surface point clouds, once the ground point cloud extraction fails, the state estimation error will increase. Lio_sam is an extended version of lego_loam which solves the least-squares problem by continuously adding various factors. However, loop detection requires a high correlation of correct data, and incorrect matching can lead to the failure of the whole SLAM system. The method we propose incorporates the advantages of all the above algorithms. First, the front end adopts the ideas of homogeneous model assumption from loam and extraction of ground features from lego_loam, respectively. However, we improve the Kalman filter model in LINS by the RANSAC method for point cloud ground extraction and propose an iterative Kalman filter model to eliminate the error caused by mismatching in order to achieve a tightly coupled LiDAR-inertial odometry. Secondly, the back end draws on the idea of factor graph optimization in lio_sam to construct a cost function for bit-pose graph optimization by adding ground constraints and loop detection constraints through the selection strategy of keyframes and further improve the accuracy of state estimation by solving the maximum a posteriori problem. Therefore, our proposed method performs well under three different perceptual degradation environments.To further analyze the localization and map-building effects of the proposed algorithm, the effects of the ground constraint and loop detection back end pose graph optimization modules were analyzed separately.Z-axis drift during the positioning and map-building process and the construction of a globally consistent point cloud map.The results of the experimental analysis show that the ground constraint of the back end pose graph optimization in the system had a great influence on the map building results, as shown in The loop detection of the back end pose graph optimization in the system had a substantial impact on the construction of the map, as shown in To explore the real-time performance of the proposed algorithm, we conducted experiments on the self-selected dataset and the KITTI dataset and analyzed the running time of the main modules for the down-sampling, pre-processing and scan-matching front end steps; on the back end, the running time was analyzed for the map construction, loop detection and pose graph optimization steps. The details are shown in Combined with the system framework diagram , we can (1)A tightly coupled LiDAR-IMU fusion SLAM algorithm for perceptually degraded scenes is proposed. The robustness for such special scenarios is enhanced by constructing an iterative Kalman filter at the front end and optimizing the pose map at the back end. By adding loop detection and ground constraints to the tightly coupled frame, the relative poses between adjacent keyframes are optimized, the cumulative sensor errors are reduced, the accuracy of the system is improved and global map consistency is ensured.(2)The algorithm was quantitatively and qualitatively analyzed on public and self-selected datasets and compared with existing LiDAR-SLAM algorithms. The results show that the algorithm was able to achieve robust 6DOF state estimation and high-performance SLAM in the test environments and output motion trajectories, global maps and state estimation after pose graph optimization.(3)Ablation experiments were performed on the algorithm in terms of ground constraint and loop detection to determine their impacts on the entire system and to explain the rationality of the algorithm. The running time of each major module on the front end and back end of the system was statistically analyzed for the different datasets, and the proposed algorithm was able to meet the real-time performance requirements.(4)In subsequent work, research on multi-sensor fusion based on LiDAR, vision and IMU will be conducted in conjunction with other practical scenarios of perceptual degradation to further improve the positioning accuracy and robustness of the system. In addition, the structure of the stored point cloud map data, which is the basis for the map-based positioning and navigation of the robot, was studied and is the key to ensuring real-time performance.


In the context of Internet technology, the integration of information technology and education is a powerful supplement to the traditional teaching model of higher education. Online learning has become the new development direction of the education industry in the network era. To address the problems of serious difficulty in completing online teaching tasks, difficulty in monitoring teaching effects, and fragmentation of course resources in universities, a multimodal music knowledge graph is constructed. A personalized learning strategy based on users' interest is proposed through the mining of online education data, and a music online education system has been developed on this basis. To improve the recommendation accuracy of the model, an embedding propagation knowledge graph recommendation method based on decay factors is proposed. The model considers the changes in the strength of user interest during the intra- and interlayer propagation of the knowledge graph interest map and focuses on higher-order user potential interest representations for enhancing the semantic relevance of multihop entities. The experimental results show that the proposed model brings a good prediction effect on several benchmark evaluation metrics and outperforms other comparative algorithms regarding recommendation accuracy. With the promotion of the “Internet+ education” model and the development of artificial intelligence, big data, cloud computing, and other technologies, online music education in colleges and universities, as an important means of education informatization, provides a convenient learning platform for learners beyond the restriction of time and space. And learners can access rich learning resources according to their learning interest and needs , 2. OnliHowever, current online education platforms in the universities are not perfect in terms of function, focusing only on teachers' design and support for learning content, with relatively little attention and support for students' participatory learning and creative activities. The technical design level is still tainted with the old disease of traditional teaching methods and thinking. The development model is not conducive to the construction of an inquiry-based learning environment. It also has a bad impact on teacher-student interaction and creative talent development . MoreovePersonalized course recommendation for online education platforms is also a hot research issue. Current course recommendations are mainly based on collaborative filtering and data mining methods . The litAlthough several of the above methods can use low-dimensional vectors of users and items for interaction computation, multihop relationship mining is achieved by employing knowledge mapping. However, the wide range of user-item interaction types, in reality, makes the potential semantic information of higher-order interactions may be completely different from the original item representation. Moreover, as the level of propagation increases, the higher-order interaction information becomes more ambiguous. To solve the problems, this paper presents a decay factor-based interlayer interest propagation method based on the knowledge graph recommendation method of embedding propagation. It is used to solve the problem of reduced semantic relevance of knowledge that exists during interlevel propagation by focusing on higher-order user potential interest representations. In addition, to alleviate data noise and degradation problems, a purification network with residual blocks is designed to capture the focus of user interest and improve the accuracy of model recommendations.This model focuses on the strength variation of user interest during intra- and interlayer propagation of the knowledge graph interest map layers and is simulated using decay factorsThe relevance of multihop entity semantics is improved by enhancing the higher-order user potential interest representationThe innovations and contributions of this paper are as follows:This paper consists of four main parts: the first part is the introduction, the second part is the methodology, the third part is the result analysis and discussion, and the fourth part is the conclusion.Multimodal knowledge graphs are constructed with multimodal semantic relationships between entities and entities in multiple modalities, which are based on traditional knowledge graphs. A knowledge graph in higher education can accurately characterize the connections between various types of knowledge points and their associated teaching resources and play the role of modeling and managing subject resources. In addition, knowledge graphs in universities can also construct learners' learning information and accurately characterize learners' cognitive state. Combining learning resources and learners' learning information, a college education knowledge graph can generate a variety of intelligent educational applications for learners and promote personalized development.The music online education knowledge graph is a natural language processing task that fuses information and resources from the music domain to extract music knowledge contained in the metadata, audio, video, images, and text and then structures and semantically represents it according to the online education platform model defined by the music domain ontology. The music knowledge graph correlates entities at all levels of music works, music concepts, music content, music characters, and music resources, supporting unified access to music knowledge and profound knowledge discovery.This paper constructs a multimodal music knowledge graph for online education in higher education. The information in the graph covers auditory, visual, and textual. The music knowledge comes from various data sources such as audio, sheet music, chants, and metadata. The fused knowledge types include external descriptive information and music content information. It supports knowledge retrieval and knowledge discovery based on music content, as shown in Knowledge extraction is the process of extracting the knowledge contained in a data source by employing identification, understanding, etc. Music content data is a unique object for knowledge extraction in the music domain. The three main types of music content data, audio, score, and vocals, correspond to different knowledge extraction methods.(1) Audio Knowledge Extraction. Audio knowledge extraction is the process of audio-based music analysis. The general audio content analysis process includes signal preprocessing, audio feature extraction, and music content classification and inference. Audio signal preprocessing can reduce the total amount of processed data, eliminate irrelevant and interfering information, and enhance audio robustness. Audio feature extraction is the basis for all music content analysis, which analyses and extracts semantically rich information from the audio signal to obtain a compact, machine-processable feature representation. From technical analysis, low-level feature descriptions for music content include the following five categories: statistical, technical features extracted from audio data; timbral characteristics; tonal features, including the relationship between pitches in the signal; intensity-related features, such as loudness; and time-related features, such as rhythm and beat position.(2) Score Knowledge Extraction. Score knowledge extraction is a notation-based process of music content analysis. It is based on formal and structured encoding. MIDI, MusicXML, and MEI are the three most common types of score encoding. Knowledge of musical scores is processed using musical feature analysis tools such as Symbolic, Humdrum toolkit, and music21, which are used to obtain musical notation feature data related to pitch, melody, chord intervals, rhythm, instrument configuration, and voice parts. Similar to the audio-based knowledge extraction process, the symbolic feature data and machine learning techniques can also be used for deep knowledge acquisition, such as musical style analysis and anonymous composer identity analysis.(3) Choral Knowledge Extraction. Choral knowledge extraction can be referred to as a textual knowledge extraction process. This process also requires the use of natural language processing techniques. However, knowledge extraction of sung words needs to be supported by audio analysis techniques. The knowledge extraction tasks that can be accomplished using song lyrics as the object of knowledge acquisition include (1) singing structure analysis: the convolutional neural network is trained to accurately identify the repetitive structure in the self-similarity matrix encoding. It is also able to predict the position of the sung segments and mark different passages of the sung words at the corresponding positions in the audio. (2) Theme distribution detection: combining data training and manual annotation, a theme judgment model based on lyrics can be obtained. The theme model is then used to analyze the lyric data and obtain information on the theme distribution of the songs. (3) Lyrics summary extraction: based on the strong correlation between lyrics and audio, the audio scaling map technique was used to find representative fragments of the music. This is then combined with graphical and thematic analysis methods to form a summary of the lyrics. (4) Emotion description: by training an emotion recognition model, it is possible to determine the emotional tendencies expressed in the lyrics (positive or negative) and even to describe more explicit emotions.Knowledge fusion is a high-level knowledge organization that enables the integration of heterogeneous data from different knowledge sources under the exact framework specification. The steps to achieve this include ontology construction, entity alignment, entity linking, and ultimately the fusion of data, information, methods, experience, and ideas to form a high-quality knowledge graph.(1) Ontology Construction. The music online education ontology build is the knowledge framework of the music knowledge graph. It is a critical step in the fusion of music knowledge, detailing the types of conceptual entities and the types of relationships between entities, setting the knowledge specification for the subsequent extraction process of education data. The ontology construction in this paper uses a manual construction approach.The manual construction approach is combined with guidance from domain experts to artificially define constraints related to concept entity types, relationship types of concept entities, etc. (2) Entity Alignment. Entity alignment is taking entities from different data sources and corresponding them to the same entity to which they collectively refer. An essential task in knowledge fusion in the music domain is to complete the entity alignment of core entities such as musicians, musical works, and musical instruments.String similarity-based entity alignment is where text or metadata in the form of cues associated with entities are analyzed for similarity. In most cases, the entity names and entity descriptions obtained by the music knowledge graph integration are ambiguous. There is no consistent representation form or identifier for the work names and artist information involved. Furthermore, multiple versions of musical works are prevalent, with different versions of the score, performance orchestration, performance venues, rips, transcriptions, etc., all producing different audio versions and corresponding titles in multiple languages and grammatical expressions. For this reason, it is tough to align musical works and musicians solely based on string similarity in the process of automatic entity alignment, which requires inference using all available information, followed by a review and correction process involving human intervention.(3) Physical Links. While ontology construction and entity alignment complete the internal knowledge fusion of the knowledge graph, entity linking links disambiguated entities to external authoritative knowledge bases to achieve knowledge fusion between the knowledge graph and external data sources. dBpedia, as a cross-domain comprehensive knowledge base, plays a pivotal role in the interconnection of the knowledge graph. In the construction of music knowledge graphs, entities such as musicians, musical works, musical instruments, and musical concepts in dBpedia are often selected as entities to be linked. The LinkedBrainz knowledge graph is the preferred entity linking object for building the knowledge graph of music research because of the richness of music-related entities such as musical works, musicians, records, and singles. By linking to it, it can also be extended to retrieve the audio characterization data provided by AcousticBrainz, including pitch, rhythm, timbre, and other information.In the Music Knowledge Graph, music knowledge retrieval can be achieved directly by constructing query statements using the SPARQL language. In addition, this can be achieved by natural language forms and example-based knowledge retrieval. Among them, relevance and similarity retrieval based on musical examples is a unique knowledge discovery approach in the music domain, which belongs to music content-based retrieval. This type of retrieval requires deep semantic processing of music content data by the knowledge graph. For audio data, audio feature extraction techniques are used to obtain content feature data, which is then organized and stored according to the audio analysis class ontology. For score-encoded data, an RDF transformation of the score using the score-related ontology is required. Through the profound fusion of musical knowledge of different representation types in the construction process, retrieval and discovery of examples and target entities across resource types can be achieved. For example, feature analysis of an audio paradigm can lead to discovering a target entity with the same or similar feature values, which can be audio or sheet music. Music knowledge inference is the inference of unique relationships between entities or new properties of entities within entity relationships.A knowledge map of online music education with a fusion of multimodal resources can help learners choose more targeted learning resources and better learning styles. In teaching scenarios, many multimodal course resources are generated to help students understand relevant knowledge points. Still, the current course resources often stand alone and are not well connected to the knowledge points, failing to play the role of multimodal resources in interpreting the knowledge points. At the same time, the overall framework of course knowledge is mainly presented in the chapter table of contents in the book. In contrast, the relationships between the subtle knowledge points and knowledge points contained in the chapters are not very clear, making it difficult from the student's perspective to make connections between what has been learned before and after, and accepting knowledge remains difficult. Multimodal course knowledge mapping can effectively solve the above problems and assist in modeling course knowledge by mining a large number of course resources in authoritative textbooks and Internet platforms. In addition, the music knowledge map can integrate multimodal learning contents for learners and recommend relevant learning resources for them according to their learning progress, which can, to a certain extent, stimulate learners' enthusiasm and improve their learning efficiency. At the same time, using the rich semantic relationship representation between entities in the academic knowledge graph can help improve the problems of data sparsity and cold start in traditional recommendation algorithms and further enhance the quality of the recommendation system. Therefore, a personalized recommendation model for music online education courses is proposed  in the knowledge graph, the embedding representations hr and tr of the head and tail entities mapped from the relationship space are obtained by embedding representation learning.Wr is the transformation matrix for mapping from entity space to relationship space and the scoring function gr  calculates the deviation in the relationship space corresponding to the two entities.The graph embedding layer obtains embedding representations of relationships and entities at the triad level, learning item, and user representations with structural information at a fine granularity, improving the representation of users and items in the model and improving recommendation performance. In this paper, the TransR algorithm is used to learn embedding representations for multimodal knowledge graphs. The algorithm enables different entity types to be mapped into the same relational space by modeling entities and relations in different embedding spaces. Given the knowledge items  rating indicates, the greater probability that the triad is a fact. The algorithm is trained to take into account the relative order between positive and negative example triads, using a pairwise ranking loss function.G is the knowledge graph, the negative example triples are generated by random substitution, and σ (·) is the sigmoid activation function.The smaller the rating function gr . The tail entity node embedding representation qi that joins the scaling-aware attention mechanism is derived from the product of the tail entity embedding representation eit and the attention weights.e00 is the initial representation of the user. eih is the embedding representation of the head entity. ri is the embedding representation of the relational entity. The attention network function f (·) calculates the attention weight of the tail entity eit which reflects the importance of the head entity to the tail entity nodes, and the scaling-aware attention function is defined as follows.Assume that the e00, eih and ri by a splicing operation, M and b are the parameters to be learned, different subscripts indicate different layers, and finally, the coefficients of the triplet are normalized using the SoftMax function. The attention network function f (·) is as follows:The scaling-aware attention network splices together qi of the tail entities in each layer of the triple yields the embedding representation Eol of the user (item) at layer l of the propagation.Aggregating the attentional embedding representations Bol is the total number of triples in layer l. The embedding of each layer represents an effective representation of the details of the propagation process, which can improve the expressiveness of the recommendation model.In the real world, user interest is not static. It may decay over time because they are tired of browsing too many similar items. Therefore, the model is not stable when propagating between multiple layers of user interest graphs. The interest information propagated needs to show a pattern of decay over time.μi and DuL represents attenuation factor and interlayer user interest, respectively.To simulate the propagation characteristics of user interest decaying, this paper designs a method for propagating interest based on a decay factor, where user interest exhibits an overall decaying characteristic when propagating between layers.BoL, a relevance probability pi is assigned by comparing the items v of user interactions with the head entity h and the relationship entity ri in each triple.ri and h can reflect the similarity between items and entities in the association space through relevance probabilities, in which the similarity between items and their corresponding entities varies across relationship conditions. The tailed entity t in the triplet with the corresponding relevance probability pi weighted summation yields a higher-order representation of the user's potential interest ZuL, defined as follows:The knowledge graph recommendation method based on embedding propagation needs to pay attention to the problem of reduced semantic relevance during higher-order propagation when learning user and entity representations. To overcome this problem, this model uses user latent interest propagation in the higher-order propagation process to enhance the higher-order representation of users. Given the set of highest-order user triples DuL and the higher-order potential interest representation ZuL:α and β denote the weighting coefficients, and the values taken in this paper are α = 0.5 and β = 0.9.The end-user interest representation for interlayer propagation is a weighted average of the interlayer user interest representation TuL cannot yield good recommendations.Although the model has enhanced the representation of user interest for interlayer propagation, the algorithm's predictions using This is because the user interest representation suffers from data noise and degradation when the depth of the propagation layers and the depth of the propagation path is too large. To overcome this problem, this paper designs an augmented network with residual blocks to capture the focus of the current interest representation, reduce the noise caused by different paths, and thus improve the user interest representation.Hu1, Hu2, ⋯, Huk.TuL is the directly mapped part of the residual block and δ(·) represents the residual network. Specially, set Hu1 = Tu1.As shown in C(·) denotes the linear mapping function.The model then stitches together the enhanced user interest representations and projects them to obtain the final user vector.eu and ev are the final representations of the user and the item, respectively. The inner product is used to predict the user-item preference score. The score size represents the probability of user interaction with the item.π(·) is the cross-entropy loss function. p+ indicates the positive user interactions. p− indicates negative user interactions. The final loss function of the model contains the loss of the knowledge graph embedding part and the recommendation part, i.e.,λi is the equilibrium hyperparameter, λ1 = λ2 = 0.5.For the optimization problem of the model, negative interactions were randomly selected from the user's unobserved interactions to ensure accurate experimental results. And, the sample sizes of positive and negative interactions were the same. The effectiveness of the model was evaluated using a cross-entropy loss function.In this paper, experiments will be conducted on two mainstream datasets, Book-Crossing, a book dataset, and Last.FM, a music dataset, respectively . These tD is chosen to be 64. The hierarchical propagation depth L is 4. And, the number of hierarchical triples is chosen to be 128. The model was optimized using the Adam optimizer. The model parameters were initialized using the Xavier initializer. The size of the augmented network on the book and music datasets was set to 8 and 12.The values of the main parameters of the proposed model are referred as the range of experimental parameters in the mainstream recommended models. The optimal parameter settings were finally derived by analyzing the relevant features of the dataset for experimentation in the experimental phase. The training set is batch processed. The learning rate is set to 1–4. The batch size is set to 128. The number of training rounds is set to 100. The increase in embedding vector dimension, hierarchical propagation depth, and the number of hierarchical triples brings exponential computational effort to the model. The recommended results do not increase much, and even exceeding a certain threshold brings the opposite result. Therefore, the embedding vector dimension Due to the high correlation between entities and clicked items within a certain number of layers of the user interest graph, the interest of users is high. After a certain number of layers, the correlation between entities and clicked items decreases and user interest decays faster. The distribution of decay factors is not equivalent or satisfies exponential growth. The design method for the attenuation factors in the model uses a nonequivalence approach. It is assumed that user interest decays at different rates at each level of user interest and that the corresponding decay factors are independent. The attenuation factor is therefore set as a hyperparameter with initial values conforming to normal distribution.F1 and AUC2 metrics were chosen to evaluate the model during the CTR click-through rate prediction task. Recall@K and Precision@K metrics were used in the Top-K recommendation task, where the values of K were 5, 10, 20, 50, and 100, respectively. Ten experiments were conducted for all experiments with random initialization. All experiments will be conducted 10 times with random initialization, and the mean value of the 10 experiments will be taken as the final experiment result.To verify the effectiveness of the proposed model, n.K value is set to 8 and 12 on the book and music datasets, respectively. As the value of K increases, the performance of the model gradually rises. While the K value is above a threshold, the model performance decreases rapidly.To investigate the effect of the augmented network size on the performance of the model, experiments were conducted on 2 datasets with different augmented network sizes To verify the contribution of the intralayer interest propagation method based on the scaling-aware attention mechanism, the interlayer interest propagation method based on the fading factor, and the augmented network based on the residual blocks to the model, ablation experiments were conducted on 2 experimental data to demonstrate the advancement of the three. The experimental results are shown in Figures To further validate the performance metrics of the model, the experimental results of the proposed model are compared with other state-of-the-art models such as the Ripplenet model of literature , KGAT moF1 on the two sparse datasets, significantly better than the other comparable models. Compared to the state-of-the-art baseline model, the proposed model improved the AUC metrics by 3.36% and 2.49% on the Book-Crossing and Last.FM datasets, respectively, and the F1 metrics by 6.73% and 3.18%, respectively. Specifically, compared to the model of literature [F1 values of the model of literature [The CTR prediction results of the present model on the two datasets obtained after several experiments are shown in terature  and the terature  incorporterature  and thisterature , 20, 21 terature  performsterature  and liteK items with the highest match for each user. Figures 9K = 20 as an example, the proposed model improved the Precision@20 metric by 7.6% and 16.91% on the Book-Crossing and Last.FM datasets, respectively. The Recall@20 metric by 7.6% and 16.91%, respectively, fully validated the effectiveness of the proposed algorithm in this paper.In the Top-K recommendation task, all experiments are performed after the model has been trained to recommend the In this paper, a multimodal music knowledge graph applicable to online music education in colleges and universities is constructed based on an indepth analysis of the characteristics of music knowledge graphs. To improve the learning efficiency of music online education, a personalized course recommendation method based on the embedded propagation knowledge graph is designed by mining the interest data of online education users. The proposed model analyses users' learning duration and ratings of courses, predicts users' preferences, and improves the accuracy of personalized course recommendations in online education systems by using user interest enhancement techniques. The experimental results on 2 datasets well validate the effectiveness of the model. The interaction between users and items in the model is still not tight enough. In the future, it is planned to combine user social networks into the model in this paper. The knowledge graph is integrated with user information to generate a heterogeneous information network containing user information and item information to better model the interaction between users and items. The structural-semantic information is explored to enhance the accuracy and interpretability of the recommendation system.


The stock market is an important part of the capital market, and the research on the price fluctuation of the stock market has always been a hot topic for scholars. As a dynamic and complex system, the stock market is affected by various factors. However, with the development of information technology, information presents multisource and heterogeneous characteristics, and the transmission speed and mode of information have changed greatly. The explanation and influence of multi-source and heterogeneous information on stock market price fluctuations need further study. In this paper, a graph fusion and embedding method for multi-source heterogeneous information of Chinese stock market is established. Relational dimension information is introduced to realize the effective fusion of multi-source heterogeneous data information. A multi-attention graph neural network based on nodes and semantics is constructed to mine the implied semantics of fusion graph data and capture the influence of multi-source heterogeneous information on stock market price fluctuations. Experiments show that the proposed multi-source heterogeneous information fusion methods is superior to tensor or vector fusion method, and the constructed multi-attention diagram neural network has a better ability to explain stock market price fluctuations. As an important part of the overall financial market, the stock market has been widely studied by scholars and investors. Stock price trends are nonlinear and unstable time series, and scholars have extensively studied stock price fluctuations. In the past 30 years, China’s stock market, as the most rapidly developing emerging capital market, has become the focus of scholars’ research to prevent financial risks and obtain investment profits. In stock market research, scholars have proposed various theories to explain the volatility mechanism of the stock market, among which the most famous are the efficient market hypothesis and behavioral finance theory. The efficient market hypothesis  states tFirst, the feature extraction of news text is not sufficiently comprehensive. Information on emotional factors , 12, objTo address these challenges, we propose a graph fusion approach for multisource heterogeneous data and construct a graph neural network based on nodes and semantic attention mechanisms for predicting stock market price volatility trends. Specifically, we construct a complex network of multiple sources of heterogeneous information in the stock market to achieve heterogeneous data fusion of trading indicator data and stock market news text. A graphical neural network model based on multiple attention mechanisms is constructed to mine the interaction between multiple sources of heterogeneous information and the combined effect of information spillover on the stock market to predict stock market price fluctuations. The experimental results of stock market volatility prediction and strategy backtesting show that the proposed method achieves good results in stock market prediction.In summary, the main innovative contributions of this study are as follows:We efficiently quantify multisource heterogeneous information in the securities market, highlighting the financial characteristics of indicators. The multisource heterogeneous information is divided into heterogeneous nodes, and the information granularity is refined. Quantify the emotional features of news texts and set node features to integrate rational and irrational factors.Innovatively introduce the relational dimension to build a complex network of time series. Based on effectively realizing the fusion of multisource heterogeneous information, it improves the efficiency of news mining in the securities market, solves the multiscale problem of financial time series, and interprets the correlation of time series.We design a graph neural network with nodes and semantic attention as the core to capture the impact of information spillovers formed by interrelated multisource heterogeneous information on the price fluctuations of the securities market. Overcoming the limitations of traditional econometric models for nonlinear and complex financial data analysis, the deep learning model constructed in this paper is a very meaningful exploration of contemporary financial research.It has been proven in stock market research that stock price trend prediction is closely related to the characteristics of financial time series . In factIn recent years, scholars have studied and explored the fusion of multi-source heterogeneous data in the stock market. Zhang  proposedGraph neural networks were developed to solve the deep learning problem of graph data. In just a few years, graph neural network technology has advanced considerably and has experienced wide application –39. BrunIn recent years, some efforts have been devoted to designing graph neural network learning for heterogeneous graphs. Wang  et al. pBased on previous research, this paper constructs a complex network based on multiple sources of heterogeneous information in the Chinese stock market, realizes effective data fusion, and introduces nodes and a semantic attention mechanism graph neural network to capture the interaction between multiple sources of heterogeneous information and the comprehensive impact of information spillover on the stock market. The proposed method is described in detail in the next section.In this paper, a graph neural network model based on multiple attention mechanisms is proposed for stock market volatility prediction analysis. The model constructed in this paper defines stock market trading data and stock market news as different types of nodes and combines the characteristics of different types of nodes based on edges, weights of edges and information transfer direction using GRU for subgraph node aggregation, which is designed with a parallel structure to improve model efficiency. The aggregated vertices form the heterogeneous graph data, and the corresponding attention mechanisms of homogeneous and heterogeneous nodes are constructed by combining node semantics to accurately express the message overflow brought by information correlation between different semantics. Finally, the heterogeneous graph is classified to predict the stock market trend. In https://aiqicha.baidu.com) and qichacha (https://www.qcc.com). The process of news text data collection on the internet is shown in As stock market news text is unstructured data, the acquisition and quantitative representation is the key process of feature extraction of multisource heterogeneous data. In this paper, considerable effort is devoted to this process, including data acquisition and text quantification. To obtain more comprehensive news text data, this paper uses the Python language, builds a lightweight web crawling engine to complete the task, uses web crawler technology to simulate browser previews of the news of the listed companies to be studied, and obtains the required information through scripting programs. To improve the efficiency of information acquisition, the theme crawling method is chosen to crawl the text information directed to the news information on the specific web pages of aiqicha  for the classification of each news headline classified into the four classes. The output vector is used as the sentiment quantization feature of the news subgraph nodes in the next section.The heterogeneous graph G =  consists of two types of indicator node sets and connection sets. The meta-paths are shown as red arrows in In stock market volatility prediction, each type of indicator node presents different importance on the meta-path, and the attention mechanism for the nodes is based on the heterogeneous node neighborhoods constituted by the path. The attention mechanism construction process is shown in Ci space for the original node However, for the two different types of nodes with heterogeneous node features, the feature space of heterogeneous nodes is first converted. The node feature conversion is performed using GRU, which is shadowed to a unified feature Attention1 is used mainly for identification under a φ meta-path, i.e., under a neighborhood. The importance of different types of nodes to each other is expressed by the weights formed by the attention mechanism.The attention mechanism for nodes φ coefficients under the softmax is introduced to perform the normalization operation. The activation function in this paper is LeakyReLU because the previous operations are linear. The function increases nonlinear factors, thereby improving the efficiency of mining useful information from complex networks. Another role is to limit the data expansion and alleviate the memory overflow phenomenon when actually processing data. ‖ denotes the feature connection operation.φ, the vertex nodes j∈φ node features Based on meta-path odes. Eq  reflectsIn view of the small-world and law-distribution characteristics of the complex network of the stock market constructed in this paper, to stabilize the training characteristics of the model and reduce the impact of excessive variance of the graph data, the model uses multiheaded attention and repeats the attention mechanism construction operation K times, as shown in Eq . FinallyAttention2 meta-paths. As shown in Eq  and qichacha (https://www.qcc.com). The total number of texts between January 1, 2016 and December 31, 2020, for the 300 stocks is 1807207 . The database news texts were classified as domestic finance, overseas finance, securities market, Asia-Pacific stock market, European and American market, individual stock news, and industry information . The data used in this paper are obtained from the CMSAR database (China Stock Market & Accounting Research Database) of CSI 300 constituent stocks from January 1, 2016, to December 31, 2020. The stock news data used in this paper are from the CMSAR database and internet crawling, and the internet crawling was performed via aiqicha , i.e., the number of labels is set to 4 classes. num_epoch = 1000: the number of epochs is set to 1000 to improve the training efficiency. batch_size = 10, and learning_rate = 5e-5. The training process parameters are set as follows: TOTAL_ITERATIONS = 1000 , VALID_ITERATION = 50 , DELAY_NUM = 100 , USE_L2 = True (a regularization method is applied), and L2_LAMBDA = 0.04 (the regularization parameter is set to 0.04). The BERT model achieves an accuracy of 85.59% after training, and all feature extraction training and testing processes are conducted using data from 2016 to 2019 to ensure the objectivity of the subsequent experimental results. The test set used to validate the data fusion and strategy backtesting experiments was not within the scope of the feature extraction experiments. The 2020 news text is only output as the result of the feature extraction method, and the feature vector is the sentiment quantification of the news subgraph nodes.A confusion matrix of the results is shown in The performance of the proposed stock market data fusion analysis forecasting method is measured according to references , 23–25 bTo further verify the superiority of the constructed model when applied to massive news text information, this paper introduces the multisource metric methods SVM, RandomForest, MKKM , and TeSThe experimental results in To demonstrate the value and practical effect of the proposed method for stock market risk warning and quantitative investment, we select the CSI 300 constituent stocks in the test set from 2020-01-01 to 2020-12-31 for backtesting. If the model predicts a rise, it generates a buy signal; a fall generates a sell signal: successive identical signals do not trigger trading operations. The initial capital of the strategy is 10,000, and the closing price of the trading day is used as the return settlement standard. The returns of the trading strategies constructed by the model are presented. Figs Based on the open world hypothesis, this paper constructs the complex network of stock market to realize multi-source heterogeneous data fusion. Node and semantic attention mechanisms are introduced to construct a graph neural network, and fully mine the node features and meta path semantic information of heterogeneous networks to complete the prediction of stock market price fluctuation. The multi-source heterogeneous information fusion and prediction method proposed in this paper is superior to the traditional method and extends the application of graph data and graph neural networks in the research of stock market price fluctuations. Although the method proposed in this paper improves the explanatory power of multi-source heterogeneous data information on stock market price fluctuations, there are many factors that affect stock market price fluctuations. We will try to expand the increase in the next step research trading data derivative index, graphical data, such as rich and complex network node types and features, further study of the interaction between the multi-source heterogeneous data information mechanism, and exploration to build a public opinion monitoring and information mining function of securities market risk prevention and control regulation system.S1 Data(DOCX)Click here for additional data file.


Electrical impedance tomography (EIT) is a radiation-free and noninvasive medical image reconstruction technique in which a current is injected and the reflected voltage is received through electrodes. EIT electrodes require good connection with the skin for data acquisition and image reconstruction. However, detached electrodes are a common occurrence and cause measurement errors in EIT clinical applications. To address these issues, in this study, we proposed a method for detecting faulty electrodes using the differential voltage value of the detached electrode in an EIT system. Additionally, we proposed the voltage-replace and voltage-shift methods to compensate for invalid data from the faulty electrodes. In this study, we present the simulation, experimental, and in vivo chest results of our proposed methods to verify and evaluate the feasibility of this approach. Electrical impedance tomography (EIT) is a radiation-free and noninvasive medical image reconstruction technique, in which a current is injected and the reflected voltage is received through electrodes . EIT canTraditional EIT systems usually assess the internal electrical conductivity of the lungs using adhesive electrodes attached to the skin of the thorax . HoweverSentec AG Switzerland has commercialized textile electrodes for EIT ,33,34. HEIT reconstructs resistivity distributions inside the body by injecting a current and receiving the reflected voltage signal from the collected voltages using 16 electrodes attached to the surfaces of the body ,37. TherTo address these problems, in this study, we proposed a method for detecting faulty electrodes using the differential voltage value of the detached electrodes in EIT systems. Additionally, we proposed the voltage-replace and voltage-shift methods to compensate for invalid data from the faulty electrodes. Our proposed methods estimate the detached electrodes and compensate for invalid data based on the measured voltages. Because our proposed methods would not require a modified data acquisition scheme or a modified reconstruction algorithm, these methods have low computation complexity. In this study, we present the simulation, experimental, and in vivo chest results of our proposed methods to verify and evaluate the feasibility of this approach.In this study, we estimated the conductivity distribution by determining current stimulation patterns and boundary conditions using a forward model of EIT. The forward solutions could be computed using the finite-element method (FEM) by discretizing the domain. Triangular elements were chosen for use in the EIT model, and the edges of the elements are vertex nodes s in EIT . The volequation :(2)Y(σ)q follows :(3)u=HσT images ,45,46.In this study, the bipolar adjacent stimulation pattern was used for our portable 16-electrode EIT system. Major changes in conductivity occur on the object boundary because of the intensity of the injection current in the EIT system . AdditioOne circular inclusion with low conductivity was simulated using EIDORS for a 16-electrode EIT system with an adjacent stimulation pattern, as shown in We simulated one sample image of a circular inclusion with low conductivity, as shown in We proposed two methods, the voltage-replace and voltage-shift methods, to compensate for spike data caused by the current injected by the detached electrode.The voltage changes in each electrode are correlated with the adjacent electrodes. Thus, we proposed the use of the voltage-replace method to use the measurement voltage values when the adjacent electrode injects a current to directly replace the measurement voltage values when the disconnected electrode injects a current and is set as the ground. In Difference EIT calculates the conductivity distribution between the measured voltage values and the reference voltage values. Therefore, when we use the voltage-replace method to compensate for the invalid data from faulty electrodes, we must replace these data in the measured voltage data and the reference voltage data. Here, the reference voltage data indicate the background data acquired from a homogeneous medium in the acrylic phantom or unnecessary chest movements during full exhalation in human subjects.We proposed another voltage-shift method, which is the modified voltage-replace method, to compensate for invalid data due to the faulty electrode. In In this example, the fifth electrode is detached in the EIT system. In the voltage-shift method, the third 16 points  are shifted by one point. Thus, the value of point 33 is shifted to point 34 and the value of point 48 is shifted to point 33. The sixth 16 points  are shifted fifteen points. Thus, the value of point 81 is shifted to point 80 and the value of point 96 is shifted to point 95. Finally, the third 16 points  that were shifted one point replaced the fourth 16 points  and the sixth 16 points  that were shifted fifteen points replaced the fifth 16 points .In the receiving signal part, 16 amplifiers  were used to measure differential voltages between two adjacent electrodes . The output voltages of the differential amplifiers were digitized by applying them to a 12-bit and 1-volt analog-to-digital converter ADC . In this study, the Zynq-based board was used to communicate with MATLAB software using the JTAG interface.We used EIDORS to simulate a cylinder tank model containing 16 electrodes with adjacent injection patterns. One acrylic rod with a diameter of 4 cm was simulated and placed on three locations at , , and . We reconstructed 3 EIT images using 16 electrodes with the first, seventh, and fifth electrodes detached, respectively. Then, we compensated for the invalid data using our proposed voltage-replace and voltage-shift methods to reconstruct EIT images again.Additionally, our experimental setup used a cylindrical acrylic tank (14 cm in diameter) filled with conductive saline, and the tank was mounted on one layer of 16 EIT electrodes a. One acThe electrical conductivity distribution of acrylic rod imaging was reconstructed from the background and measurement data. Here, the background data indicate the received signal coming from homogeneous scattering mediums only in the acrylic tank, and the measurement data indicate the received voltages from the acrylic rod placed in the acrylic tank.We evaluated the figures of merit of the EIT image and presented the potential for performance improvements after compensating for invalid data using our proposed methods. We used position error (PE) and shape deformation (SD) to quantitatively analyze the reconstructed EIT images from the simulation and experimental data ,49. We cThe in vivo EIT data of the chest in healthy volunteers were acquired using the wearable textile electrode belt shown in Here, we reconstructed lung images using difference EIT and calculated the conductivity distribution between the reference and measurement voltage values. Thus, we asked the volunteers to fully exhale to record the background data. Then, the background data were processed to a 208-voltage dataset and served as the reference voltage values. The measurement data were collected during natural breathing. After that, the measurement data were processed to a 208-voltage dataset and served as the measurement voltage values.We used the structural similarity index measurement (SSIM) to quantitatively analyze similarities in brightness, contrast, and structure between the original EIT lung image and the faulty EIT lung image or the compensated EIT lung image . The SSIA block diagram of the 16-electrode EIT system with compensation for invalid data is presented in We found that the results of the EIT images obtained from the EIDORS simulation  were simThe EIT images of the chest  over one breathing cycle using a portable EIT system with the wearable textile electrode belt are shown in In this study, we proposed a method for detecting faulty electrodes using the differential voltage value of the detached electrode in EIT systems. Additionally, we have verified the validity of our proposed voltage-replace and voltage-shift methods to compensate for the invalid data when an electrode of an EIT system is detached in the simulation and experimental results. Furthermore, our proposed methods were applied to in vivo lung data using a portable EIT system with a wearable textile electrode belt. The in vivo results have verified the feasibility of our proposed methods.


Bipedal robots have gained increasing attention for their human-like mobility which allows them to work in various human-scale environments. However, their inherent instability makes it difficult to control their balance while they are physically interacting with the environment. This study proposes a novel balance controller for bipedal robots based on a behavior cloning model as one of the machine learning techniques. The behavior cloning model employs two deep neural networks (DNNs) trained on human-operated balancing data, so that the trained model can predict the desired wrench required to maintain the balance of the bipedal robot. Based on the prediction of the desired wrench, the joint torques for both legs are calculated using robot dynamics. The performance of the developed balance controller was validated with a bipedal lower-body robotic system through simulation and experimental tests by providing random perturbations in the frontal plane. The developed balance controller demonstrated superior performance with respect to resistance to balance loss compared to the conventional balance control method, while generating a smoother balancing movement for the robot. As robot technologies advance, various types of mobile robots, such as wheeled, legged, and tracked robots, are being investigated for a variety of applications. Many robotic applications are designed for human-scale environments, in which both robots and humans can physically coexist. A bipedal humanoid robot is considered the prime candidate for robotic tasks in human-scale spaces because its shape and dimensions are the same as those of the humans for which the space was designed. However, although remarkable advances have been made in humanoid technologies, humanoid robots with human-level mobility and manipulability are still a distant prospect. As demonstrated at the DARPA Robotics Challenge held in 2015, state-of-the-art robotic systems struggle with simple physical tasks that their human counterparts can easily perform in daily life [Whereas a conventional industrial robot performs tasks on a fixed base, a bipedal humanoid robot is an underactuated multibody system on a floating base. Owing to the inherent instability of floating-base robots, it is difficult to control the balance of these humanoids while they perform locomotion and object manipulation.Humanoid balance control has been investigated in two categories: walking balance ,3 and stFor standing balance control, force control methods are known to have several advantages over position control, by directly controlling the contact forces and moments by utilizing the low impedance of the humanoid mechanism even with a relatively small number of sensors. Several studies have been conducted based on passivity theory ,7,8,9,10These methods using the simple feedback law and the MPC set the desired wrench to be proportional to the difference between the current states of the robot  and a fixed reference configuration (FRC). However, this approach is rather simplistic compared to the balancing control employed by humans. In human motor control based on a human’s own experiences, complex neural activities allow the human to maintain balance by changing the reference configuration pertaining to the human’s surrounding environment, rather than by using an FRC.Humans, unlike their humanoid counterparts, have the innate ability to maintain balance by shifting their CoM. Human balance control is carried out by combining various balance strategies . AnticipRecently, artificial intelligence (AI) technologies have demonstrated capabilities that approach or exceed the human level, in various fields of application. Machine learning is a subfield of AI, in which AI algorithms learn from training data to make predictions or decisions. Among machine learning techniques, deep neural networks (DNNs) have shown superior performance in handling nonlinear problems with high robustness against noise in the fields of computer vision , speech Reinforcement learning and imitation learning are commonly used to train DNNs for robotic applications. Reinforcement learning algorithms with well-designed reward functions have been shown to be effective for locomotion applications ,28,29. HThis study developed a novel balance controller for a bipedal robot based on machine learning techniques and robot dynamics. Behavior cloning, one type of imitation learning, is used to mimic human balancing strategies to recover from unknown force perturbations. The unknown force perturbations considered as force applied to the bipedal robot by environment while object manipulation. To cope with the perturbations, the desired wrench at the CoM is computed using a behavior cloning model trained by human demonstrations of balancing. To achieve the desired wrench at the CoM, the joint torques are calculated based on the robot dynamics. In this study, each perturbation was modeled as the internal force applied by the upper body on the lower body, while the upper body performed manipulation tasks that required physical contact with the environment. The performance of the balance controller was validated using a bipedal lower-body robotic system by simulation and experimental tests under random perturbations in the frontal plane. The results show that the developed balance controller is capable of generating a smooth balancing motion against various types of perturbations, compared to the conventional balance controller.The remainder of this paper is organized as follows. The humanoid under balance control can be modeled as two parallel-coupled subsystems: the upper and lower bodies. The upper body is modeled as an impedance system for manipulation, and the lower body is modeled as a floating base system for balancing ,34. The The external force can be considered to have the same effect as the equivalent internal force and moment at the interface between the upper and lower bodies c, as if To collect human demonstration data, we used a teleoperation system in which a human operator controlled the balance of a bipedal robot with the approval of the Deliberation Committee (KUIRB-2020-0277-01). While collecting the data, an actual robot and a robot simulator were interacted with the human operator in a human-in-the-loop manner to maintain the balance by using a custom-made HMI. The custom-made HMI, Balance Feedback Interface (BFI), was developed by the MIT Biomimetics Robotics Laboratory . The BFIThe actual robot, Little HERMES , and the simulator for Little HERMES were used to acquire the data and evaluate the performance of the developed balance control system. The design of Little HERMES has several features that simplify its robot dynamics. The bipedal robot has spherical feet, which makes each foot of the robot contact with the ground at a single point. At a single point of contact with the ground, the GRF has only three directional force components without any moment components. Each leg has three actuated degrees of freedom for hip flexion/extension, hip abduction/adduction, and knee flexion/extension. Six actuators for the two legs were placed in the body trunk using timing belts for the actuation of the knee joints, as shown in The sensors implemented in the actual robot include an inertial measurement unit (IMU) and six joint encoders. The IMU attached to the body trunk was used to measure the orientation and three-directional angular velocities of the body trunk. The joint encoders measure the joint angles of the two legs. The CoM states of the robot can be estimated based on measurements from the IMU and the joint encoder.We developed a dynamic simulator for Little HERMES. In the simulator, a multi-body dynamics model of Little HERMES was programmed using the physics engine MuJoCo . The modUpon the application of an external force on the bipedal robot, the change in the balance state of the robot is reflected by the force and moment applied by the two arms of the BFI on the human operator. The feedback force and moment are applied on the trunk of the human operator in the frontal plane (red arrows in WH) contains two force components (x- and z-axes) and one moment component (y-axis). The two force components at the CoM are equivalent to those of the resultant GRF measured by the force plate. The moment at the CoM can be estimated based on the two force components of the resultant GRF and the relative position between the CoM and the CoP, as described in Equation (2). In the frontal plane, the net wrench at the CoM of the human operator  and the DCM of the human operator (ξH) are scaled to provide the reference wrench (Wref) and the reference DCM (ξref) for the bipedal robot. The reference DCM (ξref) is then compared with the measured DCM of the bipedal robot (ξR). The difference between the reference DCM (ξref) and the scaled DCM of the bipedal robot  to yield the desired wrench at the CoM of the bipedal robot (WD). The computation of the desired wrench (WD) to maintain the balance of the bipedal robot is expressed as in Equation (4).As illustrated in al robot c is multrobot  and robot (i = R)With the desired wrench at the CoM  at least one of the robot’s feet lost contact with the ground, and the robot was no longer in the double support phase; (2) the DCM moved outside the support line formed between the two feet of the robot in contact with the ground; and (3) the angle between the midline of the robot trunk and the vertical line in the frontal plane exceeded 20°. If any of the three conditions were met, the bipedal robot was considered to have lost its balance, and data acquisition was halted.DCM Net and W Net.The balance controller was trained to control the balancing motion of the robot using the human-operated balancing data described in the previous section. ller see a. The wrThe structure and hyperparameters of the DNN should be carefully determined before training the DNN using human demonstration data. Among the various DNN structures, long short-term memory (LSTM) is known to be suitable for handling time series data because it can model complex nonlinear feature interactions ,40. HoweDCM Net and W Net were selected by following the process described in DCM Net and W Net  layers, respectively. DCM Net uses ReLu activation function for its six FC layers and a linear activation function for its output layer, whereas W Net uses Tanh activation functions for all FC layers and the output layer. During the training process, the Adam optimizer was used to update the weights of DCM Net and W Net. The learning rate (LR), the decay rate of the LR (DR), and the batch size (BS) were set to 0.000719, 0.005219, and 1024 for DCM Net and 0.002875, 0.009715, and 128 for W Net, respectively.The optimized MLP structures for DCM Net, and W Net. The three candidates for the DNN structures were evaluated with the actual robot using a test dataset that was not used for training. DCM Net to estimate the reference DCM  layers, whereas W Net has four pairs of C-P layers. In both networks, the output of the last C-P layer was transformed into a 1D vector using a flatten layer to be later used for the FC layer. Both DCM Net and W Net have nine FC layers after the last C-P layer. During the training process, the RMSprop optimizer was used to update the weights of DCM Net and W Net. The LR, DR, and BS were set to 0.0001, 0.000001, and 1024 for DCM Net and 0.0001, 0.000001, and 1024 for W Net, respectively.The CNN-MLP structures for BCBC) was compared with that of the conventional control based on the FRC (FRCBC). With the robot dynamic simulator, the pattern of the external force, including the amplitude, pulse width, and time interval, can be easily modified. The performances of BCBC and FRCBC were evaluated in terms of the resistance to balance loss and the jerkiness of the balancing motion.In the robot dynamic simulator, the performance of the balance controller using the behavior cloning model (BCBC and FRCBC at which the foot slip (blue bars) and the loss of foot-ground contact (red bars) occurred when a single pulse  and (7) when a single pulse was applied with varying amplitudes from 21 to 27 N and a pulse width of 100 ms. BCBC were lower than those with FRCBC for both linear and angular balancing movements in the horizontal plane for the pulse amplitudes of 21 N and 22 N. Particularly for the angular motion, the jerk cost functions with BCBC were lower than those with FRCBC by an order of magnitude. As described in the third column of FRCBC lost its balance with a pulse amplitude above 23 N, whereas with BCBC, the balance loss occurred with a pulse amplitude of 28 N. The simulation results show that the robot controlled by the behavior cloning model generated smoother balancing movements for higher external forces without losing its balance, compared to the conventional FRC-based control. This feature is advantageous for improving the lifespan of the motors and other mechanical parts of the robot by reducing jerky impact loads.The results show that the jerk cost functions with BCBC and FRCBC were tested with an actual robotic system, Little HERMES. For experimental tests with the actual robot, external forces with irregular shapes were manually applied as the disturbance to balance. The performances of BCBC and FRCBC were evaluated in terms of the resistance to balance loss and the jerkiness of the balancing motion.The performances of BCBC and FRCBC. The impulse is a more suitable measure than the force since the magnitude of the manually applied force is difficult to control.In the robot dynamic simulator, the resistance to balance loss was evaluated using the amplitude of the external force at the balance loss. In experimental tests with the actual robot, we used the impulse  with which foot-ground contact was lost, rather than the external force itself, to compare the performance of In the experiments, impulsive forces were manually applied to the trunk of the robot. The impulse of the force was increased while its magnitude was monitored until foot-ground contact was lost. We performed 10 sets of experiments, and each set of experiments included five cases of foot-ground contact loss. The minimum value of impulse out of the five cases was selected as the threshold impulse that initiated foot-ground contact loss for each set of experiments. From the 10 sets of experiments, 10 threshold impulses were selected.BCBC and FRCBC. The results show that the threshold impulses with BCBC  was much higher than those with FRCBC , with a statistically significant difference (p < 0.05). These results further support the fact that the behavior-cloning model can improve the capability of the robot to maintain balance compared to the FRCBC against irregularly shaped external forces.The bar graph in BCBC (blue bars) and FRCBC (red bars). In the figure, the averages of the jerk functions were compared in four different ranges of the impulse magnitude: [1500, 2000), [2000, 2500), [2500, 3000), and [3000, 3500). The robots equipped with BCBC and FRCBC were able to maintain its balance with impulse magnitudes up to 3500 N·s and 2500 N·s, respectively. For the jerk cost function of linear motion, BCBC showed slightly larger cost values than FRCBC in the ranges of [1500, 2000) and [2000, 2500). As for the jerk cost function of angular motion, however, BCBC showed lower cost values than did FRCBC for the same ranges. The experimental results show that the behavior cloning model can generate angular motion with higher smoothness, and linear motion with comparable smoothness, compared to the conventional FRC-based control.The jerk cost functions for the linear and angular movements were computed for the balancing motion of the actual robot. In this study, we developed a novel balance controller for bipedal robots in the frontal plane by employing a behavior cloning model, as one of deep learning techniques. Unlike FRC-based studies on robot balancing, the developed control algorithm generates the desired wrench at the CoM by capturing human balancing behavior from the robot states without referring to a specific reference frame.DCM Net and W Net, were trained on human-operated balancing data acquired using the BFI. The two DNN structures after training the map from the measured robot states to the desired wrench at the CoM of the robot. The joint torque controller calculates the joint torques from the desired wrench based on the robot dynamics model of the bipedal robot. This structure allows the developed balance controller to cope with various contact conditions, including changing the contact points and time-varying contact forces.The developed balance controller consists of two components: a wrench estimator and a joint torque controller. The wrench estimator calculates the desired wrench at the CoM using a pre-trained behavior cloning model that employs two separate DNNs. The two DNNs, The performance of the developed system was evaluated experimentally using an actual robotic system, as well as in simulations using a robot dynamics model. Both the simulation and experimental results show that the developed controller outperforms conventional FRC-based balance controllers with respect to the prevention of foot slip and foot contact loss under various types of external perturbations. The results also show that the developed balance controller can generate smoother angular movements compared to conventional FRC-based controllers. These results confirm that the balancing controller developed in this study outperforms the conventional FRC-based balance controller by moving the reference configuration based on human balancing movements, while employing the same PD-controller as in the conventional balancing method.As a preliminary study, we investigated the balancing control of a bipedal robot in the frontal plane. Although balancing in the sagittal plane involves different balancing mechanisms, the behavior cloning-based control methodology developed for frontal plane motion can be easily extended to achieve control balance in the sagittal plane. In our future work, we plan to apply the balance controller to a bipedal robot performing power manipulations in the sagittal plane, such as hammering and axing.