At present, virtual-reality fusion smart experiments mostly employ visual perception devices to collect user behavior data, but this method faces the obstacles of distance, angle, occlusion, light, and a variety of other factors of indoor interactive input devices. Moreover, the essence of the traditional multimodal fusion algorithm (TMFA) is to analyze the user's experimental intent serially using single-mode information, which cannot fully utilize the intent information of each mode. Therefore, this paper designs a multimodal fusion algorithm  which focuses on the parallel fusion of user's experimental intent. The essence of the MFA is the fusion of multimodal intent probability. At the same time, this paper designs a smart glove based on the virtual-reality fusion experiments, which can integrate multichannel data such as voice, visual, and sensor. This smart glove can not only capture user's experimental intent but also navigate, guide, or warn user's operation behaviors, and it has stronger perception capabilities compared to any other data glove or smart experimental device. The experimental results demonstrate that the smart glove presented in this paper can be widely employed in the chemical experiment teaching based on virtual-reality fusion. Many secondary school chemistry experiments currently contain damaging, costly, and sometimes dangerous experimental qualities, preventing students from doing practical operations for many of them. Simultaneously, because teachers have a finite amount of teaching energy, some students are prone to overlooking the most important aspects of experimental operation and exhibiting irregular operation behavior when conducting experiments. As a result, developing an intelligent, operable, and low-risk experimental platform system for secondary school experiments has become a pressing issue.Three popular and effective technical solutions are available. The first is to create a virtual experimental platform  using viIn light of the shortcomings of the previous three technical solutions, this paper finds that just one or two sensors can be employed to detect the user's hand position and movement in the study of the smart glove. As a result, this paper designs a smart glove and a mixed reality (MR) experiment system that can fuse multimodal data and, using a multimodal fusion algorithm, determine the user's experimental intent and direct the experiment, as well as correct and remind the user of any incorrect or unsafe procedures. The experimental intent of this paper refers to the experimental steps of user operation. By obtaining the real-time experimental intent of the user, the smart glove system can advise the user in time or correct and remind the user of incorrect and harmful steps. And the foundation for completing the experiment is understanding the user's experiment intent.This paper designs a smart glove system based on MR experiment teaching, to overcome the limits of existing interaction devices and tools. Using multisensing fusion technology, the smart glove can complete complex user experiment intent analysis using only some simple sensors. In this study, a monocular camera is integrated at the wrist of the smart glove to solve the problems of visual occlusion and observation of subtle phenomena. Not only can the device capture data about the user's behavior throughout the experiment, but it can also detect the user's experimental intent and demand for the experimental scene.Under the background of intelligent experiment, this paper proposes a multimodal fusion algorithm (MFA) which fuses the user's experimental intention in parallel at the intention level, and solves the limitations of the traditional multimodal fusion algorithm (TMFA) which is used to analyzing the user's behavior in serial. After acquiring the intent probability in the user's voice, visual, and sensor channels, this paper employs information weight method to fuse the user intent probabilities. The essence of MFA is to convert the user's abstract intent which is difficult to understand into a calculable probability problem.Therefore, the following are the primary innovations in this paper:This paper is organized as follows. The virtual-reality integration experiment is an experiment in which users interact with the virtual world using real-world experimental objects to interact with the virtual objects in a computer-simulated virtual environment.Virtual experiments were first proposed by William Wolfe in 1989, and as computer multimedia graphic picture technology advanced, virtual experiments were gradually included in educational training. At the beginning of the development of virtual experimental teaching mainly based on Web technology, Aljuhani et al.  developeVirtual-reality technology has progressively become popular as research progresses. Bogusevschi et al.  used virIn comparison with genuine trials, the virtual-reality fusion experiment not only increases users' interest in learning and aids their comprehension of knowledge, but also reduces consumables and risks. Virtual-reality fusion studies, as opposed to virtual experiments that divide the real from the virtual, provide an interactive feedback pathway between the virtual world, the real world, and the user, increasing the realism of the user experience. Researchers are gradually mixing MR technology with experimental education instruction, and MR technology has therefore become a popular study direction because it possesses the features of virtual-reality fusion and real-time interaction.Originally, multimodal fusion meant combining various senses. The use of more than one input channel  to communicate with a machine in a system is referred to as multimodal interaction. As a result, adopting multimodal input interaction in virtual-reality settings might improve the naturalness and efficiency of interaction compared to using unimodal .Ismail et al.  merged vIn short, the multimodal fusion interactive approach can solve the problems of input incomprehension, incompleteness, and misunderstanding caused by unimodal interaction with the system, as well as the ambiguity caused by relying on only one modality's input information to understand the user's intent. Existing multimodal fusion algorithms, on the other hand, primarily use a variety of single-channel information to assess user intent serially, but they are unable to fuse multichannel information to analyze user intent in parallel.The interaction method of keyboard and mouse combination has the attribute of having a weak sense of genuine operation in the field of human-computer interaction. Using traditional indoor interactive input devices for behavior detection, on the other hand, will result in the problems of occlusion and inability to observe subtle experimental phenomena. As a result, the above two human-computer interface methods each have their drawbacks. However, the advent of the smart glove gives a more natural human-computer interaction tool. It can interact with real objects using its presence of sensing devices and is not limited by the camera's field of vision, and it has responsiveness, good real-time, and high precision. Therefore, the smart glove is widely employed in a variety of applications, including sign language recognition, robot manipulation, and rehabilitation training.Lokhande et al.  developeIn the realm of robot manipulation and rehabilitation, the smart glove has also played an essential role. To enable remote manipulation of a robot, Roy et al.  designedIn conclusion, the current virtual experiment platform primarily completes experiments by playing virtual animations, and users lack real-world experience with the platform. Simultaneously, other experiments rely primarily on unimodal interaction, which lacks an effective understanding and feedback of user intent. Although a traditional data glove can collect user data, it lacks knowledge of the user's intent and perception of external information during human-computer contact, and most of them only employ a single channel for interaction. Therefore, this paper proposes a smart glove with cognitive capabilities based on multimodal fusion, in which a multimodal fusion algorithm fuses user intent from visual, sensor, and voice channels in parallel at the intention level. The smart glove system can also employ MR technology to show the corresponding experimental phenomena and achieve the functions of guidance and error correction for the user after acquiring the user's final experimental intent.This paper designs a new smart glove with multisensing fusion in the context of a secondary school experiment. This smart glove can collect multimodal data about the user and then detect the user's operational intent, which is characterized by convenience, operability, and flexibility. Flex sensor: This sensor (Flex Sensor 4.5) is positioned in the smart glove's finger section and is used to obtain the degree of bending of the user's finger. The smart glove can map the bending changes of the sensor to the virtual hand in the Unity virtual scene, which is then utilized to restore the user's finger's bending state in real time.Attitude sensor: This sensor (MPU6050) is positioned in the back of the smart glove's hand and is used to restore the user's hand's real-time posture by measuring the three-axis angle, velocity, and acceleration of the user's hand movement.Pressure sensor: This sensor is located at the end of the smart glove's finger and is responsible for monitoring the user's finger end movements during operation.Vibration motor module: The sensor is positioned in the back part of the smart glove's hand, and it can provide vibration feedback when the user grasps a real or virtual object.Monocular camera: The camera is fixed in the wrist part of the smart glove, which can obtain the information of objects in experimental scene in real time and observe the subtle experimental phenomena. Its appearance solves the occlusion problem of traditional indoor input devices when identifying objects.Indoor binocular camera for virtual-reality fusion: This camera is different from the occlusion-causing indoor camera. This binocular camera and Unity's Vuforia plug-in are used by the MR experimental system to provide a conduit between the actual world and the virtual world, allowing real experimental objects to interact with virtual experimental objects. Simultaneously, it can track the smart glove's real-time motion trajectory.Flex sensors, attitude sensors, pressure sensors, vibration motor modules, a monocular camera in the wrist part of the glove, and an indoor binocular camera applied to an MR experimental system make up the hardware element of the smart glove system. Different sensors can gather different information about the user's hands. To improve the traditional data glove which seriously affects the user's operation due to the complicated wires, this paper improves the data transmission between the smart glove and the computer to Bluetooth wireless transmission mode. The following is the functional design of each of its components.Trajectory Acquisition of Smart Glove Based on SGBM and YOLOv5 FusionDuring the experiment, the smart glove needs to acquire the user's hand movement trajectory and map it to the virtual experiment scene built by Unity in real time. The purpose is to enable the smart glove to interact with the experimental objects in the virtual-reality fusion experimental scene, which is the foundation for determining the user's experimental intent.x-axis and y-axis), but also the smart glove's depth value z, which is one of the paper's more difficult issues.This paper incorporates the indoor binocular camera's binocular range function into the YOLOv5 target recognition procedure, while the smart glove is moving. The depth coordinates of the smart glove are obtained using the binocular stereo matching algorithm SGBM, with camera calibration, stereo correction, stereo matching, and range as the primary experimental steps. This fusion technique allows YOLOv5 to obtain not only the smart glove's position in the plane direction  is the smart glove's 3D position mapped to the virtual world, and k is the coordinate transformation's scale factor. Finally, the data are transferred to the Unity platform through a socket connection.The coordinate information is processed using equation  based onThe majority of virtual laboratories are provided in the form of augmented reality (AR) or virtual-reality (VR), with participants immersed in a purely imaginary scenario of the experiment without being able to observe their actions in detail. As a result, this paper uses Unity's Vuforia plug-in and an indoor binocular camera to create an MR lab. This lab can integrate virtual sceneries and the actual world using AR technology, allowing them to cohabit and interact. MR is divided into two components in its practical application. The virtual object exhibited outside the screen is one component, and the real scene displayed inside the screen is the other.The footage from the indoor camera is combined with the virtual world, creating a new visualization of the experimental environment. This fusion process allows users in the MR lab to genuinely watch their operations as well as observe the experimental phenomena that arise as a result of those operations as shown in The smart glove system incorporates a speech recognition module to better conduct the virtual-reality fusion experiment teaching and collect the user's intent. The Baidu voice SDK is used in this study to continually monitor and recognize the user's voice information. And the MFA is used to calculate the intention probability after the effective information in the experiment is obtained; at the same time, the module uses the Unity's SpVoice plug-in to provide users with voice guidance or explain experimental phenomena, reducing the user's memory burden.To guide and correct users' experimental operations, this paper uses the cognitive-behavioral theory , which rIn the input layer, the smart glove acquires multimodal information from the user's voice, visual, and sensor channels, and passes the data into the recognition layer. The recognition layer and the fusion layer are critical components of the smart glove's overall framework. Under the background of intelligent experiment, the TMFA has the flaw of user experiment intention obtained by serial fusion of single-mode information. To overcome this challenge, the intent probability models for the voice, visual, and sensor channels in the recognition layer are established, and they are utilized to update the intent probability of their respective channels in real time. By studying user behavior, these intent probability models convert the abstract intent of user behavior under each channel into a computable set of intent probabilities by analyzing the user behavior. In the fusion layer, this study ingeniously proposes the MFA, which dynamically updates the associated weight of each channel using the information weight method, fuses the intent probability set produced by the recognition layer in parallel, and finally determines the user's experimental intent.In summary, the essence of the model and algorithm proposed in this paper is to convert the abstract intent, which is difficult to speculate, into a mathematical language that can be computed.i will be dynamically updated each time the user performs an experimental operation. The visual channel intent probability acquisition algorithm (hereinafter referred to as VSIPAA) is based on YOLOv5, as shown in Algorithm 1.The monocular camera on the smart glove can perceive the entire experimental scenario through the visual channel. To obtain the user's experimental intent under the visual channel, this model uses the incremental change of the bounding box area of the experimental object in YOLOv5 to infer the probability of the user's experimental intent. The target object objDifferent experimental items correspond to one or more experimental intent in the visual channel. When the VEIPAA algorithm updates the usage probability of the experimental items, the probability of one or more experimental intents corresponding to them is also updated in real time.γ to establish the mapping relationship among the three flex sensors, pressure sensors, and attitude sensors. On this basis, this paper designs the Euclidean distance sensor channel based intent probability acquisition algorithm (hereinafter referred to as SRIPAA) in the seven-dimensional space, as shown in Algorithm 2.Distinct experimental actions correlate to different experimental intents in the sensor channel. During the experimental operation, the user dynamically generates a huge amount of complex sensor data, so this paper sets up a seven-dimensional vector Under the sensor channel, different actions correspond to one or more experimental intents, and when the probability of the current user action is updated by the SRIPAA algorithm in this paper, the probability of the corresponding one or more experimental intents behind it is also updated simultaneously in real time.i into verb sv and nouns sn using Baidu voice recognition and lexical analysis technology, and put them into the task slot to get the instruction V, as shown in V performs similarity matching with the system speech library SPEECH task slot.Users can enter speech information at any point during the experiment in the voice channel. This study divides speechUnder the voice channel, this paper designs a voice channel intention probability acquisition algorithm based on task slot (hereinafter referred to as VCIPPA), as shown in Algorithm 3.Multimodal fusion refers to the overall merging of all input information before the system confirms the user's intent in the context of a virtual-reality fusion experiment. The TMFA completes the experiment by applying the rule that one channel's input information corresponds to one experimental intent, which essence is that the serial integrates the diversity of single-mode information, rather than the parallel fusion of all modal intent at the intention level.Therefore, this paper inventively proposes MFA based on the smart glove system, which calculates the coefficient of variation of each channel using the information weight method and obtains the weights of each channel by normalization, and then the joint intent probability of multimodal fusion is calculated and generated and achieves the function of fusing user multimodal intent information in parallel on the intent layer in the following steps:Algorithm validity means that when the input value satisfies the condition, the algorithm should work properly and output the corresponding result; i.e., whether the breakpoint is set in the voice channel, the visual channel or the sensor channel will output the corresponding intent. In the real experiment, the user may not input the information of three channels at the same time; for example, the user may not input the voice during the operation. In that case, the algorithm will update the coefficient of variation of the visual and sensor channels in real time and flexibly to calculate the final intent probability.When the user perceives the experimental scene information using the smart glove's camera on the wrist, the MFA employs YOLOv5 to identify the experimental objects in the scene and calculates the increase or decrease of the area of the experimental object between two frames in real time. This algorithm feeds the area increment change value of the experimental object's bounding to the VSIPAA algorithm, and the probability set Visual of the experimental intent is obtained under the visual channel.γ to establish the mapping relationship between the three sensors, so the intent inference problem of the sensor channel can be converted into a problem of calculating the distance between vectors in the high-dimensional space. To acquire the experimental intent probability set Sensor under the sensor channel, the MFA utilizes SRIPPA algorithm to calculate the Euclidean distance of the vector in the high-dimensional space.When users use the smart glove to operate real or virtual experimental objects, the smart glove will continuously generate a large amount of unavailable and independent complex data in the process of operation, so the SRIPAA algorithm sets up a seven-dimensional vector V. By matching with the similarity of system speech library SPEECH, the intent probability set Voice under the voice channel can be easily calculated.Under the voice channel, users can input different speech information speech at any time. And the MFA utilizes VCIPAA algorithm to generate input speech task slot Throughout the experiment, the MFA changes the value of each channel's intent probability set in real time, as well as the corresponding weight value, i.e., the coefficient of variation, in response to the change in the probability value. Finally, based on the normalized coefficients of variation, the MFA performs an intention-level fusion of the probability sets of the three channels to produce the user's final intent.The MFA efficiently handles two theoretically suggested problems in this paper: (1) the MFA provides an unobstructed real-time perception of scene information using the visual channel at the wrist of the smart glove compared to the indoor interactive input devices; (2) the MFA can fuse the user's multimodal intent in parallel at the intention level and improves on the TMFA serial processing of modal information.The computer used to conduct the research in this paper had an i7-10875H processor with a 2.30 GHz processor and 16 GB of RAM. The virtual-reality fusion lab was designed using Unity with version number 2018.3.8. To evaluate the effectiveness of the smart glove system with multimodal fusion in improving the teaching of MR experiments, 30 volunteers were invited to participate in the experiments while designing the comparison experiments in this paper. The volunteers included 10 elementary school students and 20 secondary school students with a male-to-female ratio of 1 : 1. The ages of these students were concentrated between 8 and 16 years old, and none of them had ever used the smart glove MR experiment system.I1), pour distilled water (I2), check air tightness (I3), pick up the clarified lime aqueous (I4), pour clarified lime aqueous (I5), pick up the spoon (I6), take out the charcoal powder (I7), add charcoal powder (I8), take out the iron oxide powder (I9), add iron oxide powder (I10), and turn on the alcohol burner (I11).In this paper, a multimodal fusion of smart glove was utilized for the charcoal reduction of iron oxide experiment. And the system speech library of this experiment is shown in Meanwhile, according to the experimental requirements, this paper sets up the experimental intent library to analyze the user's experimental intent, as shown in The smart glove system receives the user's multimodal data in real time and converts it into the intent probability set of each channel throughout the user experiment operation. Next, Algorithm 4 performs a parallel fusion of intention levels on the intent probability set, before finally displaying the user's current experimental intent. As illustrated in In this experiment, the second step is to pour distilled water. The user wears a smart glove, pours liquid from a real beaker into a virtual beaker, and inputs the voice “pour distilled water.” Through Algorithm 4, the system deduces that the user's present behavior intends to “pour distilled water.” The MR experiment system gives real-time feedback on the user's current behavior through the experiment scene animation, as shown in V and performs similarity matching with the system speech library, so as to update the intent probability set of the voice channel. Although users can input varied speech information into the smart glove system at any time, the intent probability of voice channel will change only when they input speeches similar to the system speech library. Consequently, to verify the stability of the voice channel, a comparative experiment is set up and 10 volunteers are invited to input the 11 instructions of the system speech library and irrelevant instructions, and the consequences of their recognition accuracy are tallied, as shown in The smart glove system can evaluate various speech information of users for voice input. The VCIPAA algorithm translates the user's speech information into instruction Instead of wearing smart glove throughout the recognition process, the user can only input speech information, preventing information from other channels from interfering with the verification of voice channels. It should be highlighted that a successful recognition instruction indicates that the MR experimental system will display the related experimental phenomenon when the user inputs the corresponding speech.As shown in One of the biggest issues with traditional indoor interactive input devices in the context of intelligent experiments is occlusion. The instruments and devices in the experimental scenario are often occluded due to different placements, making the input device unable to perceive and recognize key information, especially difficult to recognize tiny objects and observe subtle phenomena. Traditional virtual-reality fusion chemistry experiments  used KINIn this paper, 30 volunteers were invited to participate in a comparative experiment in which they completed several key steps of both the traditional virtual-reality fusion experiment and the MR experiment based on smart glove system. Each volunteer conducted 3 times, their recognition accuracy was counted , and the results are shown in As can be learned from For the action of picking up a medicine spoon, it is difficult for KINECT to respond to it because the spoon is small and not easy to recognize, but the wrist camera of smart glove system can easily recognize the object in the experimental scene. For the action of turning on the alcohol burner, the magnitude of the action is small, so the KINECT device can easily recognize the gesture as other actions. However, the smart glove system can employ MFA to fuse multichannel information to recognize the action.In summary, the smart glove system can solve the problem that it is difficult to cope with subtle actions and micro-phenomena in the traditional virtual-reality fusion experiment.Users can complete the MR chemical experiment by using a single-mode interaction with an independent voice channel or by combining two or three modes. However, during the experiment, the visual and sensor channels will continuously collect data from the user, while the voice channel will be input according to the user's choices, allowing the user to choose the number of channels to employ based on the current experimental steps.In order to verify that multimodal fusion can better identify user experimental intent than single mode, 20 volunteers were invited to complete the experiment 4 times using  voice chWhen the user uses the voice channel alone, the experiment completion rate is 97.5% since the input voice information is rather constant. Users, on the other hand, are unable to operate genuine experimental objects, which result in the user experience is terrible.Because an experimental object in the visual channel and an action in the sensor channel might both correspond to separate experimental intentions at the same time, the average completion rate drops, but the user's score improves significantly when compared to the single voice channel. Because the data from the sensor and visual channels are complicated, the temporal complexity of dual-mode fusion increases, resulting in a longer average completion time for users.Due to the nonstandard operation of the user in the multimodal fusion process, the average completion rate of the experiment will be lower than that of utilizing only the voice channel, but the user score will be the highest in the comparative experiment. Furthermore, when compared to dual-mode fusion, after the addition of a voice channel, the average completion time of a multimodal fusion experiment is lowered. So, it is shown that the experimental interaction mode of multimodal fusion can suit users' needs and can better comprehend their experimental intentions.Although the TMFA  uses mulP), effort (E), and frustration (F). The NASA assessment metrics were evaluated on a 5-point scale. 0 to 1 indicates a low cognitive burden, 1 to 2 indicates a relatively low cognitive burden, 2 to 3 indicates an overall cognitive burden, 3 to 4 indicates a relatively high cognitive burden, and 4 to 5 indicates a very high cognitive burden. The results are shown in To highlight the ability of the smart glove to perceive the behavioral intent of users, a comparison experiment was designed in this paper. First, volunteers conducted experiments on a virtual experimental platform using the Noitom data glove  and the In conclusion, as compared to existing experimental platforms, the smart glove system developed in this paper allows users to conduct experiments more intelligently and naturally, as well as improves users' experimental immersion and operation ability.Some middle school experiments currently have issues with powerful destruction and expensive costs, while the traditional experimental platform suffers from a poor sense of operation and a significant memory burden. As a result, this paper develops a smart glove using the MFA to detect the user's experimental intent and then direct the user to experiment, or correct and remind the user's incorrect and harmful actions.This study designs and implements a smart glove system. The smart glove can address the issues of (a) the lack of cognitive ability of traditional data gloves and (b) the occlusion phenomenon caused by too many indoor interactive input devices in the traditional virtual-reality fusion chemistry experiment. The traditional data glove can only gather the user's hand data and cannot capture the user's experimental intent; thus, several indoor interactive input devices are required to aid the user during the perception process of scene information in real time, which can easily cause occlusion in object recognition. However, the smart glove system can realize the observation of tiny objects, subtle movements, as well as minute phenomena, increase interaction efficiency, and decrease memory pressure. However, the smart glove's hardware arrangement is unreasonable, and it is difficult to wear when the user's palm is large; the MR system is currently just for chemical research. As a result, future research should focus on improving the smart glove's structure and expanding the experiment library so that users can complete a wide range of experiments in order to achieve the goal of virtual-reality education .A parallel MFA integrating sensor, voice, and visual channels is proposed in this paper to address the problem that the TMFA only integrates user modal information in serial. The essence of TMFA is the serial fusion of users' multimodal information, and it does not support the simultaneous analysis of users' intentions using multimodal data. However, the MFA is a multimodal fusion algorithm based on multichannel intention probability fusion, which can accurately and efficiently obtain the user's intent and guide user's behavior based on that intent. Based on the user's intent, future research should be devoted to predicting the user's intent in the following stage and building a human-computer cooperation model so that the user can wear the smart glove and cooperate with the computer to complete experiments.To address the aforementioned issues, this study primarily performs the following two original contributions:


For improving the dynamic quality and steady-state performance, the hybrid controller based on recurrent neural network (RNN) is designed to implement the position control of the magnetic levitation ball system in this study. This hybrid controller consists of a baseline controller, an RNN identifier, and an RNN controller. In the hybrid controller, the baseline controller based on the control law of proportional-integral-derivative is firstly employed to provide the online learning sample and maintain the system stability at the early control phase. Then, the RNN identifier is trained online to learn the accurate inverse model of the controlled object. Next, the RNN controller shared the same structures and parameters with the RNN identifier is applied to add the precise compensation control quantity in real-time. Finally, the effectiveness and advancement of the proposed hybrid control strategy are comprehensively validated by the simulation and experimental tests of tracking step, square, sinusoidal, and trapezoidal signals. The results indicate that the RNN-based hybrid controller can obtain higher precision and faster adjustment than the comparison controllers and has strong anti-interference ability and robustness. With the open-loop unstable and inherent nonlinear characteristics, it is a constant challenge to design the high-performance control scheme for position control of the magnetic levitation system7.Benefiting from its superiority of the contactless, frictionless, and noiseless, magnetic levitation system has been successfully applied to many fields9, proportional-integral-derivative (PID) control11, model predictive control13, robust H-infinity control15, sliding mode control17, and adaptive fuzzy control19. Although these different control strategies can achieve good control results from different perspectives7, there is still room for improvement in the control performance of the magnetic levitation system to a certain extent.Over the years, various control strategies have been successively developed to implement real-time position control of the magnetic levitation system. The control strategies mainly include feedback linearization control20. Correspondingly, ANNs can improve the position control performance of the magnetic levitation system due to its powerful self-learning and adaptive abilities. For example, Rubio et al. presented a neural network controller consisting of a nonlinear method and a neural network and reduced the root mean square error of the trajectory tracking in the magnetic levitation system21. Silva et al. constructed a neural controller to magnetic levitation system and obtained a better control performance than the classical controllers22. Wei et al. proposed a feedback compensation controller based on backpropagation neural network (BPNN), which lowered the steady-state error in position control of the magnetic levitation ball system23. Yang et al. designed an adaptive sliding mode controller (SMC) based on radial basis function neural network (RBFNN) to the magnetic levitation system and acquired faster convergence and stronger robustness than the traditional SMC method24. Sahoo et al. applied the functional link artificial neural network to the Fuzzy PID controller and validated its superiority in the real-time Maglev control system25. Tang et al. presented a hybrid controller based on BPNN and fuzzy inference for the magnetic levitation ball system, which improved the dynamical response of tracking step and square signals26. Qin et al. designed a model predictive controller based on RBFNN to control the position of the maglev ball, which obtained a better track performance of the step signal than conventional PID controller27. Sun et al. presented a supervisor control method based on the RBF neural network and achieved an excellent tracking performance in the flexible and time-delay control of the maglev train system28.Recently, the intelligent control based on artificial neural networks (ANNs) can obtain a better tracking performance and disturbance rejection, benefiting from ANNs’ excellent ability to learn the dynamic model of the nonlinear control system26. However, the parameter updating of these feedforward neural networks reckon without the historical information of the control system, which limits the nonlinear approximation of the controlled object to some extent, thereby hindering the further improvement in the control performance of the magnetic levitation system.Through the above literature review, it can be found that these neural networks used in the control systems belong to the feedforward neural networks, which can improve the control performance of the magnetic levitation system by approximating a nonlinear function without prior knowledge of the closed-loop control system30. Lin et al. designed a hybrid computed force control method with an RNN uncertainty observer, which improved the tracking accuracy in position control of the magnetic levitation system30. Fatemimoghadam et al. proposed an adaptive backstepping control scheme based on a projection RNN for the magnetic levitation system, which achieved a better control performance than the sliding mode control method31. Jafari and Hagan applied RNN to the model reference control of the magnetic levitation system and obtained better performance than linear PID controllers32. Hou et al. embedded a recurrent feature selection neural network to an intelligent global sliding mode controller (GSMC) and acquired superior performance than traditional GSMC33.The recurrent neural network (RNN) can acquire the dynamic response of the control system by using delays of an internal feedback loop, which has the ability to deal with the time-varying control system, especially in nonlinear and uncertain scenariosAn intelligent controller based on the accurate compensation control is developed to ensure both dynamical performance and steady-state performance of the control system, which consists of a PID-based baseline controller, an RNN identifier, and an RNN controller.The RNN identifier is designed to online learn the inverse model of the control system, and the learning parameters are passed to the RNN controller sharing the same structure as the RNN identifier in real-time, which achieves the accurate compensation control.The simulation analysis and experimental verification are shown comprehensively to prove the effectiveness and advancement of the proposed RNN-based intelligent controller and the improvement of the transient performance and the steady-state accuracy with a certain robustness.Therefore, motived by the mentioned superiorities of the RNN in the controller design of the nonlinear time-varying system, a hybrid controller based on RNN is proposed to improve the control performance in the position control of the magnetic levitation ball system in this article. The main contributions are summarized in the following.The remainder of this article is organized as follows. Section “This study focuses on the position control of a magnetic levitation ball system, and the schematic diagram of its physical entity is shown in Fig. To describe the position control problem of the magnetic levitation ball system, as shown in Fig. First, the steel ball is only subject to the electromagnetic force 22Then, based on Kirchoff’s law and Biot-Savart’s law, the electromagnetic force When the ball is in the equilibrium position, its acceleration is zero and can be expressed asNext, according to Faraday's law and Kirchoff’s law, the relationship between control voltage and current in the electromagnet winding can be expressed as followsFinally, the magnetic levitation ball system can be described jointly by the above equations, and the expression is as followsExpanding the Taylor series at the equilibrium point When omitting the high-order term, the kinetic equation of the steel ball can be rewritten asFor the magnetic levitation ball system, the input voltage n of Eq. , the opeSubstitute the boundary equation 2 to Eq. , then thAccording to the characteristic equation of the open-loop system, the open-loop poles of the system can be obtained as Furthermore, let Define the coefficient matrix in Eq.  as folloP and the observability matrix Q of the magnetic levitation ball system can be calculated bySo the controllability matrix P and Q can be obtained as followsFrom Eq. , the ranP is equal to the dimension of the state variables, while the rank of the observability matrix Q is equal to the dimension of the output vector. In other words, the magnetic levitation ball system is both controllable and observable. Consequently, the controller can be designed to make the system stable.As known from Eq. , the ranIn this section, a hybrid controller is designed for position control of the magnetic levitation ball system. As shown in Fig. For the hybrid controller, the PID controller is the baseline controller, and the RNN controller is the compensation controller. Where the PID controller is mainly applied to provide online samples for the RNN identifier to learn the inverse model of the controlled object in real-time and also used to maintain the stability of the control system when the RNN is untrained at the early control phase. And the RNN controller sharing the same network structure as the RNN identifier is utilized to carry out accurate compensation control.In addition, the RNN identifier is employed to learn the inverse model of the controlled object online, considering the historical data of the control system. And the learning parameters from the RNN identifier are passed to the RNN controller in real-time. Therefore, the proposed controller can achieve the high-precision position control of the magnetic levitation ball, benefiting from that the RNN identifier accurately learns the inverse model of the control object, and the RNN controller implements exact compensation control in the real-time.u(k) of the hybrid controller is designed as followsConcretely, the control law 34. Consequently, this article takes the PID controller as a baseline controller in the proposed hybrid control strategy.With the characteristics of clear functionality, structure simplicity, ease of use, and applicability, the PID controller has been regarded as the simplest and yet most efficient scheme for lots of industrial control problemsub at the moment k is defined as followsIn practice, as a baseline controller of the hybrid control strategy, the PID controller is employed to provide the online learning samples and maintain the system stability at the early control phase, especially when the neural network is untrained. Concretely, the control law As one main part of the hybrid control strategy, the RNN controller is utilized for the position compensation in the magnetic levitation ball system. Concretely, as shown in Fig. k can be defined as followsFor the RNN controller, the tracking error k is calculated as followsith weight between the input and hidden layers, ith weight between the moments k − 1 and k in the hidden layer, ith bias in the hidden layer, The hidden layer of the RNN controller at moment k is obtained as followsith weight between the hidden and output layers.The output layer of the RNN controller at moment k can be obtained as followsSubstitute Eqs.  and 17)17) into It is worthy to note that the RNN controller only carries out the forward propagation calculation but does not carry out the back propagation calculation, and the solving parameters , the baseline controller based on PID (PID), the hybrid controller based on PID and BPNN (PID + BPNN), and the hybrid controller based on PID and RNN (PID + RNN) are quantitatively compared on the following measurements, namely, the mean absolute error (MAE), the root-mean-square error (RMSE), the integral time absolute error (ITAE), and the integral time square error (ITSE). The formulations of these four measurements are expressed as followsThe tracking controls of the step, square, sinusoidal, and trapezoidal signals are implemented in the simulation tests. In particular, for the tracking controls of step and square signals, the overshoot and settling time are considered besides MAE and RMSE. For the tracking controls sinusoidal and trapezoidal signals, the steady state error is considered besides MAE and RMSE. Moreover, for the fair comparison of the different control strategies, we try our best to make control performance to be the best by adjusting the appropriate parameters of each controller. Take the tracking controls of sinusoidal and trapezoidal signals as examples, the simulation results of the RNN-based hybrid controller under different hyper-parameters are shown in Fig. Ultimately, for the simulation tests, the optimal parameters of the different controllers are described in Table For the simulation tests, the control responses of tracking step and square signals under the different comparison controllers are presented in Figs. As shown in Figs. Concretely, as shown in Table To further validate the control performances, the tracking controls of sinusoidal and trapezoidal signals are simulated, the control responses are drawn in Figs. It can be known from Figs. From Table To further verify the effectiveness and the advancement of the RNN-based hybrid controller, the hardware-in-loop platform shown in Fig. In the comparison controllers, hyper-parameters play a critical role in controller performance. Take the tracking controls of sinusoidal and trapezoidal signals as examples, the experiment results of the RNN-based hybrid controller under different hyper-parameters are displayed in Fig. Firstly, in the experiment tests, the control response comparisons of tracking continuous step and square signals under the different controllers are shown in Figs. It can be learned from Figs. As shown in Table In addition, to preferably demonstrate the control performances, the tracking controls of sinusoidal and trapezoidal signals are tested in the experiment platform, and the control responses are drawn in Figs. It can be seen that the hybrid controllers can track the reference signals better and have lower steady-state error compared with the baseline controller. Particularly, the position tracking accuracy of the magnetic levitation ball is higher than the BPNN-based hybrid controller when applying the RNN-based hybrid controller. Concretely, the performance evaluations of the tracking controls are quantitatively described in Table From Table In this section, the tracking control experiments of both sawtooth and stochastic signals with constant disturbance are first utilized to indicate the disturbance rejection ability of the RNN-based intelligent controller. The experiment results are drawn in Fig. It can be found from Fig. In addition, the tracking control experiments of both sinusoidal and trapezoidal signals with stochastic disturbance are further employed to demonstrate the disturbance rejection ability of the RNN-based hybrid controller. The experiment results are drawn in Fig. It can be seen from Fig. The magnetic levitation ball system is a typical nonlinear and unstable system. For improving its control accuracy and tracking performance, this study presents an intelligent controller based on a PID-based baseline controller, an RNN identifier, and an RNN controller. The RNN identifier is designed to learn the inverse dynamics model of the controlled object online then the learning parameters are passed to the RNN controller in real-time, carrying out an accurate compensation control. And the comprehensive simulation and experiment results show that the proposed controller can improve the transient performance and lower the steady-state error with excellent robustness compared with the traditional PID controller and BPNN-based internal model controller.To further illustrate the effectiveness and the advancement of the designed controller, some advanced control approaches of the magnetic levitation ball system are experimentally compared in this section. It is worth noticing that all comparisons are performed on the same or similar experimental platform. In addition, the different methods are not compared under the time integral evaluation metric, considering the different duration of the tracking experiment.35, fuzzy control is considered to compare with the proposed controller. The experiment results of the tracking step signal show that the fluctuation range of tracking error is  using the Takagi–Sugeno fuzzy controller (TSFC)36. The error range of the TSFC is obviously larger than that of the proposed controller, which can obtain the error range of  when tracking the step signal.First, as one of the effectively intelligent control approaches7 is employed to compare with the proposed controller. As shown in Table Then, an advanced sliding mode control method named adaptive nonsingular terminal sliding mode control (ANTSMC) based on reduced order generalized proportional integral observer (RGPIO)22 is utilized to compare with the proposed controller. For the sinusoidal signal tracking, the RMSE using NNC is 0.0472, which is obviously larger than the 0.0145 obtained by the proposed controller. Besides, the fuzzy inference-based neural network compensation controller (FI-NNCC)26 is applied to compare with the proposed controller. As shown in Table Finally, a nonlinear autoregressive exogenous-type neural network controller (NARX-NNC)As a result, it can be seen from the above comparative analysis that the proposed intelligent controller based on RNN has better control performance and tracking accuracy than other advanced control approaches to the magnetic levitation ball system.In this study, the hybrid control based on RNN is proposed to achieve the high-precision position control of the magnetic levitation ball system. The main contribution of this study is that the control quantity can be adaptively compensated with high precision by the hybrid controller without establishing an accurate mathematical model of the control system. Moreover, the innovation is that the RNN controller implements high-precision control while the RNN identifier accurately learns the inverse model of the controlled object online, considering the historical data of the control system. Ultimately, the simulation and experiment results demonstrate that the RNN-based hybrid controller can improve the transient performance and lower the steady-state error with certain robustness compared with the BPNN-based hybrid controller.The proposed RNN-based hybrid scheme has been successfully applied to the magnetic levitation ball system with a single-degree-of-freedom. And the research in this article may provide some helpful references for the application of the proposed approach in the real-time control process. In addition, the proposed hybrid controller can facilitate deployment to the second-order control system with nonlinear, time-varying, and uncertain characteristics. In future work, on the one hand, we will try to design the hybrid controller based on RNN to the multi-degree of the freedom control system; on the other hand, we will try to study the high order control problem by using an RNN-based hybrid control strategy.


Load forecasting is very essential in the analysis and grid planning of power systems. For this reason, we first propose a household load forecasting method based on federated deep learning and non-intrusive load monitoring (NILM). As far as we know, this is the first research on federated learning (FL) in household load forecasting based on NILM. In this method, the integrated power is decomposed into individual device power by non-intrusive load monitoring, and the power of individual appliances is predicted separately using a federated deep learning model. Finally, the predicted power values of individual appliances are aggregated to form the total power prediction. Specifically, by separately predicting the electrical equipment to obtain the predicted power, it avoids the error caused by the strong time dependence in the power signal of a single device. In the federated deep learning prediction model, the household owners with the power data share the parameters of the local model instead of the local power data, guaranteeing the privacy of the household user data. The case results demonstrate that the proposed approach provides a better prediction effect than the traditional methodology that directly predicts the aggregated signal as a whole. In addition, experiments in various federated learning environments are designed and implemented to validate the validity of this methodology. Since AC power cannot be stored, the power generation capacity of the power plant must reach a stable dynamic balance with the power consumption , in ordee.g., the non-linear extrapolation method (etc.) and machine learning forecasting methods  (etc.). Since a single forecasting method often has various defects, the combined forecasting method is gradually applied to the research of power system load forecasting, and its performance is better than all independent load forecasting models  , supportks (ANN) , etc.. Sg models . Chen & g models  comparedg models , Autoregg models , etc.Research on household or residential electricity demand estimation currently takes several different ideas, including evaluating and improving existing load forecasting techniques, developing and researching new methods, or a combination of the two. In addition, if the predictive model is to be applied to a wider range of electricity consumers and appliances, sufficient electricity demand data must be collected from multiple data owners, but multi-source load data may cause privacy disputes . In pracIn view of the above, most of the current household load forecasts are directly based on the total power forecast as a whole, which will cause large errors. As for the few integrated prediction methods based on NILM, there are also a series of problems such as low prediction accuracy, lack of data preprocessing, not taking advantage of deep learning, or not considering the comprehensive connection of time. In addition, the centralized model is vulnerable to external attacks, which may lead to third parties having the opportunity to obtain load data. In particular, load information may be destroyed when stored in a centralized server or during transmission.(1) This article presents a load prediction method based on federated deep learning (FedDL), which combines federated learning and BiLSTM-Attention. For all we know, this is the first study of federated learning in household load forecasting based on NILM. In this prediction method, under the precondition of guaranteeing the accuracy of the forecast, the data of the users in the house is not leaked, and the data can be isolated to ensure that the data of the users in the house is not leaked, and meet the needs of user privacy protection and data security. On the precondition of guaranteeing the accuracy of the prediction, while reducing the data security risk, it can also learn from the electrical power of other households.(2) A method based on NILM to predict user household energy consumption is proposed. The NILM technology extracts a single load pattern from the historical total load demand, avoiding the strong time dependence of a single device. Experiments demonstrate that the integrated prediction method based on NILM has better prediction results than the traditional method that directly predicts the aggregated signal as a whole.(3) Use the data decomposed by the CNN-LSTM hybrid deep learning model to test the proposed federated deep learning prediction model to validate the validity of the proposed model; In addition, experiments in various federated learning environments are designed and implemented to validate the validity of this methodology.This article presents for the first time a novel federated deep learning prediction model based on NILM. This method integrates federated learning and Bi-directional long short-term memory network-Attention mechanism (BiLSTM-Attention). The following are the main contributions:The main points of this article are as follows: Section 1 introduces the relevant background and relevant work on household load forecasting; Section 2 presents the associated theories of machine learning; Section 3 mainly proposes the federated deep learning model, and introduces the specific process of training this model. Section 4 performs the relevant training and testing of the proposed FedDL model using the data obtained based on the NILM, and analyzes the test results. Section 5 summarizes this article.Due to their powerful learning and generalization capabilities, artificial neural networks are widely used to deal with pattern recognition problems in the engineering field . This ar process , separatThe pooling function represents the pooling operation, LSTM is a typical recurrent neural network, which is particularly suitable for dealing with time-series related pattern recognition problems. T. The output parameters are the memory cell state T. The corresponding calculation equation in The input parameters are the memory cell state ory unit ; \documfunction ; \documThe BiLSTM network has the function of capturing the characteristics of the information before and after . Since iT can be found by the forward hidden layer state The BiLSTM hidden layer state The attention mechanism highlights key information through weight distribution to ensure that features are not lost in the model training process, so it can more effectively mine the long-distance data features of time series with correlation . The attT-th feature vector of BiLSTM network output, which is input into the hidden layer of attention mechanism to obtain the initial state vector Y is obtained by multiplying and summating the corresponding weight coefficient T-th feature vector, and T-th feature vector (e vector .etc.) can be exchanged among participants   , but the(1) Task initialization: the server determines the training target, parameters, and participating terminal devices, and passes the global model t represents the current training round. The participant appliance i trains the model locally and updates the parameter (2) Local model training and update: global model The training process can be organized into three steps:(3) Global model aggregation and update: The server receives the model parameters uploaded by the participant devices and aggregates them, and finally updates the global model The above training process is iterated until the convergence or termination condition training technique is satisfied. From the classical training model of federated learning, we can find that the local appliance communicates with the server through model parameters during training, so as to protect privacy.In this part, the process of each part is explained in detail. Firstly, the data is preprocessed, and the individual load of household appliances is decomposed from the total load demand through the NILM method based on the CNN-LSTM hybrid model. Then the power of individual appliances is predicted separately by the FedDL prediction model, and the predicted power of individual appliances is aggregated to form the overall power prediction. Finally, the validity of the proposed model is evaluated by some evaluation indicators.In this article, the NILM method is based on the CNN-LSTM hybrid model to decompose the total load data of the training sample and obtain the energy consumption value of individual appliances. Firstly, the CNN and LSTM are trained separately using the pre-processed data, and then the feature information extracted from each of the two networks is handled as the equal dimension using the mapping layer, and finally, the two are concatenated. For more specifics, please refer to In order to build a better neural network, a large amount of data containing various devices and responding to consumer behavior habits needs to be fed to the network. However, during model training, the local data owner may encounter the risk of privacy leakage and loss of control over the data. To address this potential risk, we use federated learning, which is in sharp contrast to traditional centralized machine learning techniques where all sample data is uploaded to a single server. The flow chart of the proposed FedDL prediction model is shown in e.g., learning rate, minibatch size) for training in FedDL, and the same optimization algorithm is used.For the purpose of illustrating, we assume that each household has the same computational power, the same hyperparameters .The specific internal structure is shown in The LSTM layer can remember or forget significant information. In principle, the more layers, the stronger the learning ability. But too many layers will lead to difficult convergence and a long training time. Therefore, in the model of this article, a two-layer LSTM is set. The number of neurons in the first layer is set as 128, and the number of neurons in the second layer is 68.(1) Mean Absolute Error (MAE): The advantage is that the calculation is simple and the description error is clear. The specific formula is shown in In machine learning, there are two major problems: classification and regression. The essence of both is to predict the input. The difference between the two lies in the type of output results. The output of the classification problem is a qualitative discrete variable, while the output of the regression problem is a quantitative continuous variable. For these two kinds of problems, there are corresponding evaluation indexes to measure the validity of the models and algorithms. As a typical regression problem, power load prediction mainly has the following evaluation indexes:(2) Root Mean Square Error (RMSE): For data whose unit measurement is too large, this operation can retain the original description unit for the data. For example, if the unit of electricity load is KW, the square of the difference will increase by an order of magnitude, which will add trouble to the model description, so RMSE is chosen to maintain the original magnitude. The smaller the original root mean square error, the more accurate the prediction. The specific formula is shown in In In this part, the integrated power is decomposed into individual device power by the NILM method based on the CNN-LSTM hybrid model, and the power of each device is predicted separately using the federated deep learning model. Finally, the predicted power values of individual appliances are aggregated to form the overall power prediction. The validity of the method is confirmed through qualitative and quantitative evaluation results. In addition, experiments in various federated learning environments are designed and implemented to validate the validity of this methodology. The programming language in the experiment is python and Tensorflow2.0 is used.In the BiLSTM-Attention model used in this article, a 2-layer LSTM is set. The number of neurons in the first layer is set as 128, and the number of neurons in the second layer is 68. The activation function is tanh, the learning rate is 0.001, Adam is selected as the optimization function, and the batchsize is 512.The data set UKDALE established by British scholars Jack Kelly and William Knottenbelt is used for the experiment . The loaFor ease of analysis without loss of generality, the experimental results shown in this article take household 1 as an example. The experimental results are shown below.Using CNN-LSTM hybrid model decomposition method to decompose the total load data of the training samples to obtain the energy consumption values of each load device, without considering the simultaneous existence of multiple electrical appliances. The load decomposition results of loading equipment for a certain three days in April are shown in In Based on the load decomposition results of nine load devices, nine load prediction models are trained respectively according to the prediction model structure proposed in “Proposed prediction model”. From To prove that the NILM-based integrated prediction methodology provides a better prediction effect than the traditional methodology that directly predicts the aggregated signal as a whole, we conducted a comparative experiment based on integrated prediction after load decomposition and overall direct prediction.To reasonably evaluate the performances of the proposed approach, comparisons with other methods have been carried out. For the different models selected in this article, related experiments are done on the basis of NILM load decomposition. The same input samples are used for training the model, and the RMSE and MAE are used as the evaluation indexes of the prediction model.As can be seen from To further validate the validity of the proposed model in a more profound way, we also test it on the other four families, where the same amount of appliances and data are selected as household 1 (All the selected models are based on NILM load decomposition). Meanwhile, the first three models in To evaluate the effect of primary parameters on the robustness of the proposed FedDL prediction model, we design a different experiment with varied local Iteration period E and the client score C that performs calculations during training. Here, the number of local iteration cycles for each round of clients is set to 5, 50, and 80; the scores for the number of clients are set to 50% and 100%. The results are displayed in As can be seen from via federated deep learning. In this prediction method, under the precondition of guaranteeing the accuracy of the prediction, the data of household users are not leaked. It can reduce the data security risk and use the electrical power of other households to meet the needs for user privacy protection.(1) By combining federal learning with BiLSTM-Attention, this work develops a household load prediction approach based on non-intrusive load monitoring (2) The NILM based on the CNN-LSTM hybrid model is used to decompose the individual load of household appliances from the overall load demand, and the power of each appliance is predicted by the decomposed power of individual appliances, which avoids the strong time dependence of individual equipment. The case results demonstrate that this NILM-based integrated prediction methodology provides a better prediction effect than the traditional methodology that directly predicts the aggregated signal as a whole.(3) This article uses the data decomposed by the CNN-LSTM hybrid deep learning model to test the proposed federated deep learning prediction model. The load forecasting result is very close to the real loads. And through comparison with other models, the superiority of the proposed method is verified. In addition, experiments in various federated learning environments are implemented to validate the robustness of this methodology.This article proposes a household load prediction method based on federated deep learning and non-intrusive load monitoring. Based on the simulation results, the following conclusions can be drawn:In this article, external influence factors, such as weather factors and humidity factors, are not taken into account in the prediction process, but these factors need to be considered in more realistic scenarios. It is interesting to extend the proposed NILM algorithm for supporting the energy management of residential integrated energy systems . Besides10.7717/peerj-cs.1049/supp-1Supplemental Information 1Click here for additional data file.


With the swift development of deep learning applications, the convolutional neural network (CNN) has brought a tremendous challenge to traditional processors to fulfil computing requirements. It is urgent to embrace new strategies to improve efficiency and diminish energy consumption. Currently, diverse accelerator strategies for CNN computation based on the field-programmable gate array (FPGA) platform have been gradually explored because they have edges of high parallelism, low power consumption, and better programmability. This paper first illustrates state-of-the-art FPGA-based accelerator design by emphasizing the contributions and limitations of existing research works. Subsequently, we demonstrated significant concepts of parallel computing (PC) in the convolution algorithm and discussed how to accomplish parallelism based on the FPGA hardware structure. Eventually, with the proposed CPU+ FPGA framework, we performed experiments and compared the performance against traditional computation strategies in terms of the operation efficiency and energy consumption ratio. The results revealed that the efficiency of the FPGA platform is much higher than that of the central processing unit and graphics processing unit. With the striking expansion of the Internet and the swift development in the big data era, artificial intelligence has been widely developed and used in the past thirty years . People'At present, the most widely used solution is to use a graphics processing unit (GPU) cluster as a heterogeneous computing system composed of general computing graphics processing unit (GPUPU) and CPU for computing acceleration. However, there are some problems in using GPU clusters. With the continuous expansion of cluster scale, colossal power consumption and heat dissipation have become the limiting factors of its scale expansion. Therefore, we need to find a better acceleration scheme with higher computing performance and lower energy consumption. A field-programmable gate array (FPGA) fully meets our high performance and low power consumption requirements as a new heterogeneous parallel computing acceleration device . As a neIn addition, FPGA developers can flexibly program hardware according to specific applications, and by comparison, the structure of GPU cannot be altered once the design is completed. Lack of flexibility makes GPUs unable to apply well to most scenarios. Simultaneously, the emergence of advanced synthesis tools, such as high-level synthesis (HLS) and OpenCL, further shortens the cycle of developing a convolutional neural network (CNN) accelerator on FPGA. In recent years, FPGA has been deployed on a large scale in the data centers of Microsoft, Amazon, Alibaba, and other companies. Nowadays, cloud computing servers only not provide deep learning computation acceleration with conventional GPU but also supply customizable FPGA plans. In addition, researchers also start to apply FPGA in the domain of edge computing such as auto pilot and the industrial robot . AlthougAt present, most researches accelerate the hardware based on the forward propagation characteristics of neural network. By analyzing the parallelism of convolution algorithm of neural network, we can adapt the flexible programming of FPGA to reconstruct the suitable framework in order to achieve optimal utilization of the parallelism of convolutional algorithm. This approach can essentially help to further improve the computing speed and efficiency.Hardware acceleration is to make full use of the inherent characteristics of hardware, use hardware computing module to replace the software algorithm run by general CPU, share the workload of CPU, and improve the overall efficiency. Typical hardware accelerations include ASIC, FPGA, and GPU. Because FPGA not only has the advantages of high speed and stability like ASIC but also has the flexibility of customizing circuit structures like a software program, FPGA has attracted the attention of most scholars .Chen et al.  proposedLi et al.  proposedKalms et al.  conducteLiang et al.  pointed Cong et al.  of the UIntel proposed the FPGA deep learning accelerator (DLA) based on Intel OpenCL . The accA field-programmable gate array (FPGA) is a kind of integrated circuit, which can be modified repeatedly to design different circuit functions . DesigneFPGA uses fine-grained parallel architecture. After determining the computing clock cycle time method, the designer can establish a more optimized output mode that eliminates the need to save data in the main memory to reduce memory reading delay.Compared with GPU, FPGA has muscular programming flexibility. At the same time, FPGA supports dynamic reconfiguration of the algorithm, which has strong reconfigurability.FPGA is much lower than GPU in power consumption and has better performance under the same power consumption, which can significantly reduce the heat dissipation problem of the processor.FPGA is widely used in more and more fields because of its high programming flexibility and powerful computing performance. Compared with GPU, FPGA has the following advantages:The CNN model imitates the structure of a biological vision system and is a feedforward artificial neural network. The core idea of CNN has three points: local perception, weight sharing, and pooling .Local perception refers to the connectivity of network parts. Every neuron is only connected with part of the upper layer neurons and senses the local part instead of the entire image. The intermediary of upper and lower neurons is called the “convolutional kernel.” Weight sharing implies that all neurons use the same weight in the same convolutional kernel, remarkably reducing the number of network parameters and making it feasible to train large-scale networks. The model can learn from a local region and then convolute the whole image with the same convolution kernel. Pooling is a kind of downsampling that will separate the input images into a few rectangular regions and outputs the maximum values for each sub-region.CNN comprises three common layers: convolutional layer, pooling layer, and fully connected layer. In general, the convolutional layer is the principal layer in the CNN model. It performs convolution operations, like the image filtering function. Convolution is a local operation that “rolls” on the feature map through a specific convolution kernel size to realize the feature extraction of the image. The pooling layer is a process of dimension reduction and extraction of the visual input image based on a biological vision system. The fully connected layer classifies the data output in a CNN model.Nin is the number of input feature maps, Nout is the number of output feature maps, Sizein and Sizeout represent the sizes of input feature maps and output feature maps, Sizekernel is convolution kernel's size, and Stride is window sliding size in the process of convolution and pooling.The CNN model selected in this paper is AlexNet model, which is the champion of ImageNet competition in 2012 . It contNif and K × K convolutional kernels, and corresponding eigenvalues are extracted and output as neuron parameters, shown in  is the value at  in the output feature map fo, wt represents the weight of the corresponding position in the convolutional kernel, in is the value at  in the input feature map fi, while kx is the x value of the kth kernel and ky is the y value of the kth kernel.Convolution is the most important part of CNN model and the most computationally intensive part, which usually accounts for more than 90% of the computation of the whole neural network. Convolution means the process of accumulating by the product of input feature maps shown in . out in the n × m convolutional kernels.When a single convolutional kernel convolutes the input feature map, due to the independent product of each pixel, the For multiple pixels on the same input feature map, the identical convolutional kernels can execute in parallel, and multiple neurons of the output feature maps can be obtained at the same time, as shown in n input feature maps and various convolutional kernels. For the exact position in these distinct input feature maps, the convolutional kernels are able to convolute in parallel.The same convolutional window on the input feature map and different convolutional kernels can perform convolution operations in parallel to obtain the values of neurons at the same position of distinct output feature maps, as shown in By combining the above four parallel characteristics of convolution computing, we can exploit the programmability of FPGA hardware to realize the ability of parallel convolution computing and achieve a better acceleration effect.There are multi-level nested loops in the convolutional module of CNN, and hundreds of convolution operations need to be carried out. If the design of parallel operation is not adopted, its time complexity will be inestimable. Generally, FPGA acceleration will consider how to optimize the loop structure. When each loop is relatively independent, the loop can be expanded to improve the degree of parallelism. Because there are many product and accumulation operations in the convolutional layer, the higher the degree of parallelism we achieve, the fewer the clock cycles, and the convolutional layer needs to complete a convolution.However, in the design, choosing different levels of loop expansion may produce different results. Further, to what extent the expanded execution units share data which affects the complexity of the generated hardware and finally impacts the number of replicas and hardware operation frequency. Firstly, we need to use the parallelism between CNN outputs to divide the convolution calculation of multiple nested loops and apply the method of loop tiling and loop expansion to construct the calculation circuit directly used in the CNN layer. Loop tiling technology transforms the program with nested loops, reconstructs the code and tiles, and reorders the iteration space to improve the data locality and block parallelism . The looBased on the previous work –10, the The control module controls the transmission sequence of data streams and communication between modules. Firstly, the control logic generates a data request signal. Under the control of the signal, the input data stream is synchronized with the system clock and updated after each cycle. Then, the input data are sent to the functional modules of each parallel or serial subsystem to complete the operation of several feature maps. Finally, after consequential feature extraction in each layer, feature classification combines multiple local feature data streams into global features and sends them to the fully connected layer module for classification.The CNN heterogeneous accelerator based on FPGA discussed in this paper was implemented by OpenCL heterogeneous computing standards and aimed to complete the hardware acceleration of CNN forward propagation computing. This section tests and verifies the accelerator by using the parameters of the trained AlexNet model and images from the ImageNet dataset and then analyzes the experimental results and the performance of the FPGA accelerator system.The accelerator function verification mode designed in this paper is CPU+ FPGA. The required functions are designed on FPGA and arranged into an accelerator structure in line with CNN operation. The operating system software of the computer sends the parameters required by the model and the image data to be processed to the FPGA accelerator via the PCIe bus. Afterward, the accelerator uses the PCIe bus again to send its running results back to the computer. The corresponding system will display running results and classification outcomes, and compare them with the benchmark solution to evaluate the performance of the accelerator. The detailed steps are as below: firstly, the input data (mainly images) will be stored on the PC, and the reverse training of the neural network model has been completed on the computer. Then, the input data and model parameters will be directly passed to the FPGA with the PCIe bus.After that, the controller of FPGA will carry out the mission to analyze the input data and AlexNet model parameter input by PC and keep them in external memory (LDDR3). When the control layer receives control instructions from the PC, the corresponding modules of the various kernels and output layers are executed to complete the computations of the neural network. The output result is returned to the PC when the forward propagation calculation is completed.We selected the FPGA hardware, the Xilinx Kintex-7 development board XC7K325T as our experimental platform, and OpenCL as the development environment. The experimental environment and the configuration information of the devices we used are listed in Here, we used the trained model CaffeNet under the Caffe framework, which is the AlexNet model , 26. TheThe images used in the test are from the ImageNet image dataset. This image dataset and corresponding category tag file synset_words.txt can be downloaded from the relevant website. The image needed to be preprocessed before testing, including image size adjustment and demean processing. The required image size was 227 × 227, and these preprocessing operations were processed by the script provided by the Caffe framework. The test method of the FPGA accelerator is to select a certain number of pictures from the ImageNet dataset for classification and record the accuracy of program classification and execution time to verify the accuracy and performance of the accelerator.In PC platform, data are stored in the inner memory and transferred automatically through data buses, which means CPU and GPU do not require extra data manipulation, but the situation for FPGA is different, and the time consumption of data transmission to FPGA board memory via the PCIe bus is essential. To eliminate these cost overheads, we implemented the scheme in which data processing and transmission take place simultaneously. It is admitted that this manipulation would make an impact on the execution performance but limited to the current architecture, most of FPGA board does not have an integrated storage, and thereby, the cost of data transmitting is deemed as unavoidable. It is noteworthy that we applied PCIe bus to complete the mission, which most devices are not equipped with, and the alternative way to do is through LAN or Thunderbolt 3.In our work, OpenCL  was chosTo compare the performance of different devices, we exploited various aspects of variables. System commands like nvidia-smi or ps were utilized to measure the behaviors of CPU and GPU. Vivado  was alsoThe CNN heterogeneous accelerator structure proposed in this article achieves the image classification function, but the batch processing operation of images is not considered in the design procedure. Hence, only a single image can be classified at a time. After classifying 100 test images (randomly chosen) in the dataset, the accurate rate reached 54%, which was basically consistent with the standard results in Caffe. To evaluate the acceleration performance of the system, an experimental comparison was carried out on three platforms: FPGA, CPU, and GPU. The batch size was set to 1 to keep the conditions consistent. The test results under the three platforms were compared, as shown in the following table. The energy efficiency ratio in In Although GPU had the shortest running time and the highest speedup, the energy consumed was also the maximum. With the same power supply, FPGA tends to have a better performance than GPU. In terms of energy efficiency, the heterogeneous acceleration implementation of CNN based on the FPGA platform discussed above was better than GPU, which also implied that FPGA performed extremely well on the balance of performance and energy consumption.Although our design of deep learning accelerator is innovative and progressive, there are some limitations in our work. The section presents such limitations and puts forward possible future works.Up until now, unlike conventional computers, most FPGA boards do not merely integrate external storage. Hence, when it requires manipulating a vast dataset, the FPGA board cannot store all of them in advance but receives each piece of it successively, which essentially led to slow down the operation speed. Hence, in the future, with the further shift in the data transmission mode, it may produce better performance because time consumption on PCIe would likely to be demolished.In the experimental evaluation, we evaluated the performance with the support of other software, and it was manifested that the software is also supposed to occupy some recourses more or less. Therefore, for the future work, it is suggested that the assessment of performance should be carried out in a more controlled and sophisticated environment such as with external gauges or appropriate virtual machines .To make a clear comparison between different schemes, we opted to use the same FPGA board as in the previous work . GeneralSeveral accelerator approaches for CNN computation based on the field-programmable gate array (FPGA) platform have shown great progresses in many areas because they have edges of high parallelism, low power consumption, and better programmability. In our research, we illustrated some important concepts of parallel computing (PC) in convolution algorithm and figured out how to realize parallel computing in FPGA hardware structure.On the basis of theoretical analysis, we applied the verification model of FPGA combined with PC to verify the acceleration function of FPGA and compared the performance of FPGA with traditional computing processors in terms of operation efficiency and energy consumption ratio. The accelerator realized a complete forward propagation process of CNN, and its convolution unit combined multiple parallel computing operations to effectively improve the overall performance.In order to verify the advantages of FPGA over traditional processors, we compared the performance against traditional computation processors related to the operation efficiency and energy consumption ratio. By combining the parallelism of CNN algorithm, FPGA significantly showed a better energy consumption ratio and efficiency than traditional CPU and GPU. For the same input image data, G GTX 1660 Ti Ultra 6G-GPU showed the best performance of 4.3 s but also with the highest power consumption of 120.0 W. Compared with the Intel i5-10400F-CPU, the speedup of our FPGA accelerator was 8.67, and the energy efficiency was 3 times that of GPU and 157 times that of CPU. At the clock frequency of 1818mhz, the peak computing rate of FPGA can reach 76.19 GOPS.Although FPGA has made some achievements in accelerating deep learning with its advantages of reconfigurability and low energy consumption, it also has some disadvantages such as hardware programming difficulties and considerable time cost of the reconstruction process. Therefore, FPGA still has a long way to go to reach the stage of more widely used deep learning accelerators. As per the current development progress, some promising research directions can be outlined as follows:At present, most of the research focuses on the repetition part of matrix operation in FPGA computing optimization , 6, but The use of fixed-point data with fewer bits can improve the acceleration performance of the model, but it also brings the disadvantage of low precision . Future Data path optimization can be divided into on-chip and off-chip data transmission. Xuechao et al.  used a sThe performance of integrating multiple FPGA chips is very promising. However, how to deal with the processing scheduling and allocation between chips is still a significant challenge. Further research work may be initiated from various fine-grained divisions and weight allocations between multiple chips.It can be predicted that deep learning has a broad development prospect in the future as a revolutionary implementation method of machine learning. Similarly, the deep learning acceleration technology based on FPGA is also expected to get better with each passing day and would finally promote the reform and development of the whole field of artificial intelligence.The proposed design in this paper focused on the parallelism of CNN model. Although it was the most popular deep learning solution, the researchers are encouraged to explore other trending models too such as the transformer, proposed by Google . The tra