Deep brain stimulation (DBS) is a promising technique to relieve the symptoms in patients with intractable seizures. Although the DBS therapy for seizure suppression dates back more than 40 years, determining stimulation parameters is a significant challenge to the success of this technique. One solution to this challenge with application in a real DBS system is to design a closed-loop control system to regulate the stimulation intensity using computational models of epilepsy automatically. The main goal of the current study is to develop a robust control technique based on adaptive fuzzy terminal sliding mode control (AFTSMC) for eliminating the oscillatory spiking behavior in childhood absence epilepsy (CAE) dynamical model consisting of cortical, thalamic relay, and reticular nuclei neurons. To this end, the membrane voltage dynamics of the three coupled neurons are considered as a three-input three-output nonlinear state delay system. A fuzzy logic system is developed to estimate the unknown nonlinear dynamics of the current and delayed states of the model embedded in the control input. Chattering-free control input (continuous DBS pulses) without any singularity problem is the superiority of the proposed control method. To guarantee the bounded stability of the closed-loop system in a finite time, the upper bounds of the external disturbance and minimum estimation errors are updated online with adaptive laws without any offline tuning phase. Simulation results are provided to show the robustness of AFTSMC in the presence of uncertainty and external disturbances. According to the regional emergence of excessive electrical activity in the brain, the International League Against Epilepsy (ILAE) classified seizures as focal , generalized, and unknown3. Focal seizures are established in a local area of the brain. They can be further classified based on the patient's awareness during a seizure attack as aware (simple) and impaired awareness (complex). In generalized seizures, both hemispheres are involved at the onset of the seizure, and awareness is impaired in most cases. The regional onset is unspecified in unknown seizures, but other manifestations are known3. These classes can also be subdivided into the motor or non-motor subsets regarding movement manifestations4. Childhood absence epilepsy (CAE) is a subset of genetic/idiopathic generalized epilepsy characterized by frequent daily absence (non-motor) seizures in children between 2 to 13 years old. The typical features of CAE include sudden cessation of activity and awareness , short time duration episodes (approximately 9 s), and 2.5–3.5 Hz bilateral, synchronous spike-and-wave discharges (SWDs) in the recorded electroencephalogram (EEG) signal during seizure episodes in the brain6. Elucidating the pathogenesis of CAE, as well as developing efficacious remedies, are the fundamental challenges that should be tackled by researchers.Epilepsy refers to a set of chronic neurological diseases typified by repetitive (at least two) spontaneous seizure attacks due to abnormal bursts of electrical activity in the brain that can bring about temporary brain dysfunction13. In accordance with experimental results, numerous computational corticothalamic models have been developed to discover the underlying mechanisms of transition from regular brain activity to SWDs25. To replicate the spontaneous SWDs in EEG recordings of rats with the absence of epilepsy, Suffczynski et al.21 developed a network model of the corticothalamic circuit in which the thalamus model consisted of the thalamocortical relay cells and reticular nucleus, and the cortex role imitated by the excitatory pyramidal cell population and inhibition interneuron population. Using these mutually interconnected neuronal populations, SWDs can be rendered by dynamical bifurcation in a bistable system. Later Rodrigues et al.23 recruited a mean-field model of the human cortico-thalamic circuit to present the transition from the inter-ictal phase to SWD. Marten et al.24 presented a modified mean-field model of human EEG activity that could explain not only the classical SWDs dynamics but also more dynamical phenomena, e.g., polyspike complexes. Destexhe8 developed a conductance-based neuronal model of the thalamocortical circuit and emphasized the pivotal role of corticothalamic feedback and GABA receptors in hyper-synchronized oscillations and SWDs patterns. Motivated by this work, a 3-neuron pattern of CAE was formulated25. According to this model, long corticothalamic loop delay and low GABAA activity of thalamic relay cells are associated with CAE. Despite GABAA deficiency, it was demonstrated that axonal myelination, which occurs by reaching adulthood, might shorten the corticothalamic loop delay and lead to outgrowing seizures.Various studies on patients with absence seizures and animal models acknowledged that the SWDs in electrophysiological recordings mainly originated from the aberrant neuronal interplay between the cerebral cortex and thalamus that constitutes the corticothalamic network. In fact, the reticular thalamic nucleus, thalamic relay nuclei, and cortical neurons form a reciprocal neuronal circuit demonstrating a significant contribution in the absence of seizure onset27. Although antiepileptic drugs are the most common prescription for CAE patients, they are ineffective in intractable cases, and long-term usage might cause side effects, e.g., liver disorders28. Neurosurgical resection, which is the removal of brain tissues contributing to epilepsy, is risky due to the generalized nature of the absence of epilepsy and irreversible consequences31.CAE might negatively impact school-aged children's social behavior, self-confidence, and education, accentuating the necessity of prompt and efficient treatment33 as well as to relieve and control the symptoms in patients with intractable seizures34. Although the DBS therapy for seizure suppression dates back more than 40 years36, the outcomes left much to be desired. A thorough understanding of the underlying mechanism by which DBS terminates the seizure is of great importance. Concerning the cardinal role of the thalamus in seizure development and inhibition, the anterior and centromedian nuclei of the thalamus have been targeted to suppress refractory epilepsy37. Despite the promising results, the mechanism of action of DBS is not thoroughly clear. From the cellular perspective, DBS can regulate the activity of GABAergic neurons located in the reticular thalamic nucleus to produce inhibitory postsynaptic potentials and induce synaptic inhibition of excitatory corticothalamic relay neurons38. It has been demonstrated that the strength and duration of absence epilepsy are weakened by amplifying the GABAergic inhibitory power39.Deep brain stimulation (DBS), the application of electrical pulses to specific brain regions using implanted electrodes, is often employed to treat movement disorders such as Parkinson’s disease (PD)46 and closed-loop control54 have been the two main approaches to address this challenge. However, the open loop control approach strictly depends on the neurologist’s experience and provides constant pre-determined stimulation on either a continuous44 or periodic42 basis, which gives rise to excessive battery usage and, eventually, battery replacement surgery56. Moreover, persistent brain electrical stimulation without considering the current state of the disease interrupts the patient’s regular activity and might cause involuntary movements57. On the contrary, a closed-loop control system monitors the disease biomarkers and automatically provides real-time variable intensity stimulations, taking into account the onset and strength of the attacks52. Motivated by these benefits, various closed-loop control DBS schemes have been designed to treat neurological and neuropsychiatric disorders62. In58, a data-driven linear state-space model in which the parameters were learned by a novel input waveform was developed to build a dynamic input–output framework for neural activities. Furthermore, a real-time closed-loop simulation testbed to control the mood in depression was implemented. In59, a population of 100 thalamic neurons model was simulated to evaluate the effect of extracellular electrical stimulation on the generated local field potentials (LFPs) in tremor conditions. An autoregressive with exogenous input (ARX) model was employed to model the relationship between the stimulus current and LFPs. To restore the power spectral profiles of tremor-free conditions, an adaptive minimum variance controller was adopted, which regulates the stimulation intensity and thereby corrects the abnormal patterns of neuronal activity. Ehrens et al.60 used a fragile stochastic neuronal network model, which entails seizure and non-seizure modes regarding the synaptic weights between neurons. A hidden Markov model (HMM) was established to detect the seizure occurrences (unstable modes) followed by a state feedback gain to suppress the seizure attacks. To suppress the motor symptoms of PD, Su et al.61 proposed a DBS framework based on proportional-integral (PI) control to track the dynamic beta oscillatory activity presented in reference signals during voluntary movement. A linear controlled auto-regressive model was employed to represent the relationship between stimulation frequency and beta band power and then was coupled with Routh-Hurwitz stability analysis to tune the coefficients of the PI controller. In63, a Radial basis function neural network (RBFNN) was used in a supervisory control algorithm as an inverse model of the computational model of PD to track the desired dynamic beta power. To track the model uncertainty and provide robustness to noise and disturbance in control of brain states using DBS, Fang et al.64 proposed an adaptive robust controller to cancel uncertainties included in a state-space brain network model. Monte Carlo simulations were performed to validate the suggested algorithm.Determining stimulation parameters such as amplitude, pulse width, and frequency is a significant challenge to the success of DBS technique. Open-loop control54. Wang et al.47 proposed a PI controller to control the high-amplitude seizure activity simulated by a neural mass model (NMM). Researchers in50, investigated an NMM representing the epileptic activities of the brain as a black box and established an auto-regressive moving-average Volterra model to approximate the black box input (stimulation)-output (EEG) relationship. Then, a model predictive controller was developed to remove the epileptic waves. Zhang et al.54 designed two closed-loop PI controllers to eliminate SWDs by automatically regulating the amplitude and frequency of DBS. To this end, a basal ganglia-corticothalamic model of absence epilepsy was approximated using the linear controlled auto-regressive model and recursive least square methods. The coefficients of the PI controllers were determined by the Routh-Hurwitz stability criterion. Despite the promising results exhibited by the mentioned control methods, the unmodeled dynamics and parameter uncertainties due to the highly nonlinear and time-varying nature of the neurological systems, external disturbance rejection, and stable function of the controller in real applications have not been entirely addressed.Myriad closed-loop control systems were developed in recent years to suppress the epileptic seizures generated by computational models of epilepsy67. In68, an implantable concurrent sensing and stimulation DBS device that relies on LFP and ECoG sensors was developed and validated in an ovine model of epilepsy. A support vector machine (SVM) classification algorithm was recruited to separate the biomarkers from the stimulation artifact. DiLorenzo et al.69 devised a seizure advisory system in connection with an implantable 16-channels intracranial monitoring to predict seizures. The preclinical tests were performed in a canine model and then implantation of 15 patients to evaluate the method.Apart from simulation studies, a handful of clinical DBS systems to suppress epilepsy have been developed in the last two decades. Responsive Neurostimulation System  is the first FDA-approved commercially available closed-loop device that provides responsive brain stimulation with respect to the electrocorticogram (ECoG) recordings through one or two depth and/or subdural cortical strip leads that are implanted at the seizure focus of focal onset seizure patients. This device demonstrated favorable outcomes on 11 out of 27 patients with epilepsy71. To achieve seizure abatement, a classical SMC was integrated with an RBFNN approximator to propel the EEG waves recorded from cortical areas with undesired dynamics to track the normal background waves49. In this work, the DBS-corticothalamic system was formulated to a canonical structure, and the system’s dynamics were approximated by RBFNN, while disturbance rejection and model uncertainties were handled by the classical SMC. The main disadvantage of the proposed controller is the asymptotic stability of the SMC due to the linear structure of the sliding surface. To tackle the stability problem, terminal sliding mode (TSM) control that contains fractional order of sliding surfaces was developed to ensure the finite-time convergence of the system trajectories to the origin72. However, TSM cannot address the chattering phenomena70 of the control input, which endangers the system’s safety in real applications. Moreover, it suffers from the singularity problem that may lead to infinite control input to guarantee the ideal TSM motion75. Qian et al.51 presented a finite-time fractional order SMC in combination with RBFNN to suppress epilepsy seizures in a well-established thalamocortical NMM. To design and stability analysis of the closed-loop system, an appropriate coordinate transformation was introduced to represent a regular form of the NMM. The main bottleneck of the work is that to transform the original NMM to the regular form and design the controller signal, prior knowledge of the NMM dynamics is necessary. To overcome the aforementioned problems , in this paper, a robust control technique based on adaptive fuzzy TSM control (AFTSMC) is developed for control of the oscillatory spiking pattern of the three-neuron pattern of CAE whereby the controller is not dependent on any knowledge of the dynamics of the system to be controlled. The main contributions of this work are as follows:25 should be represented and modeled as a three-input three-output nonlinear state delay model, and three continuous DBS pulses (control input) are designed to force the CAE pattern to track the normal state. This representation is based on the decomposition of the nonlinear dynamics of the current and delayed states of the CAE system25. Indeed, the continuous DBS pulses are automatically generated based on the MIMO model of the CAE motif. To achieve this, the unknown nonlinear dynamics of the MIMO model are approximated with the fuzzy logic system (FLS).To implement the proposed MIMO control algorithm, the dynamics of the membrane voltage of three-neuron CAE motifA fast TSM-type reaching law is embedded in the continuous nonsingular control input to accelerate the speed of the motion to the sliding surface far away from the surface in finite time. The superiority of the continuous control law is that it’s chattering-free.On the sliding surface, the fractional-integral structure of the surface guaranteed the convergence of the tracking error to the neighborhood of zero in finite time.The nonlinear dynamics of the current and delayed states of the model embedded in the control input are approximated with the FLS, and adaptive laws based on the terminal gradient descent algorithm are developed for online updating the weights of the estimator.To guarantee the bounded stability of the closed-loop system, the upper bounds of the external disturbance and minimum estimation errors are updated online using terminal-based adaptive laws without any offline tuning phase.Sliding mode control (SMC) is a powerful nonlinear robust control strategy featuring outstanding performance to compensate for the parameter uncertainty and bounded external disturbances in highly nonlinear systemsA and GABAB receptors can play an important role in the activation of spike-and-wave seizures. It has been proven that the seizure appeared in the mouse homolog by reduction of the conductance of GABAA receptors. Furthermore, both the thalamus and cortex are essential to the absence of seizures. As each cell type of the thalamus and cortex neurons contain the voltage-dependent currents necessary to define their properties, the dynamics of the electrical activity of the cortical (CT), thalamic relay (TC), and reticular (RT) nuclei neurons are represented as a conductance-based Hodgkin-Huxley type neuron model. The topology of the three-coupled network is depicted in Fig. 25:−2 is the capacitance of the membrane of CT, TC, and RT neurons and −2) are denoted by + ions, + ions, ++ ions for RT neuron, + ions, ++ ions for TC neuron, +-K+ current activated by hyperpolarization, + ions, and, −2 and −2. Detailed dynamics of the ionic membrane currents are provided in Appendix CAE is a 2.5–3.5 Hz bilateral, synchronous sudden spike followed by the wave discharges (SWDs) observed in the EEG pattern of the children during seizure episodes. SWD patterns with similar characteristics in human subjects were also reported in the experimental works conducted on animal homologs. Electrophysiological recordings of the spike-and-wave seizures in animal models showed the aberrant neuronal interplay between the cerebral cortex and the two main thalamic cells involved thalamocortical and the inhibitory neurons of the thalamic reticular nucleus. These experiments suggested that the connections of the GABAA, and GABAB currents with the value of their parameters are summarized in Table E (mv) and −2) are the reversal potential, and maximal conductance, respectively, and s is the fraction of the channels opened with the presynaptic neuron. In the dynamic equation of s, the parameter T is the concentration of neurotransmitters released with the presynaptic neuron, and the parameters B), the parameter B current. In the simulation, the values of the external current of the TC neuron and maximal conductance from the RT neuron to the TC neuron (−2) and CAE state with an oscillatory spiking pattern (−2)25.The synaptic currents   Considerdered as , the conigned by , and thered with . If the Consider the first dynamic of the tracking error with or using  is rewriBy considering the Lyapunov function tituting  in its fentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaAfter the reaching of the states to the sliding surface dered in , the conThis completes the proof of Theorem Since in the real DBS systems, the dynamics of nonlinear functions mator in  and  are  are ConsFLS with  and , a, aConsid in 29), the The term ol input  and \docIn this paper, to satisfy the sufficient controllability condition of  it was aThe simulation results of the CAE and normal states of the spiking patterns of TC, CT, and RT neurons with the tracking results of the AFTSMC for suppression of the oscillatory pattern of the CAE are provided in this section. The schematic of the closed-loop control system through adaptive DBS using AFTSMC is indicated in Fig. −2 with the decrement of the strength of the maximal conductance from the RT neuron to the TC neuron from 0.65 to 0.32 mS cm−2, the rhythm of the neurons changes from bursting or single spiking to tonic spiking. Indeed, the complexity of the dynamics of the neurons is dependent on the values of the external TC current and the maximal conductance from the RT neuron to the TC neuron. The smaller selection of the maximal conductance, the faster rhythm of the neurons in the CAE state. To calculate the tracking error, the desired trajectory is the membrane voltage of the TC, CT, and RT neurons in the normal state  is calculated in the following form:−2 and pulse width, 1 ms) was applied to the dynamic equation of TC neuron in  obtained over 15 trials of the simulation for control of the CAE state in the presence of non-regular external disturbance current applied to CT neuron using AFTSMC in comparison with the AFSMC and super-twisting sliding mode control (STSMC)70 are depicted in Fig. −2 are 0.47, 0.52, 0.63, and 0.76 mv, respectively, for the CT neuron are 0.35, 0.47, 0.50, and 0.52 mv, respectively, and for the RT neuron are 0.55, 0.83, 0.92, and 0.83 mv, respectively.Figure To evaluate the robustness of the controller to handle the effect of the time-varying uncertainty of the system parameters, the values of maximal conductance and reverse potential of the ionic currents of three neurons were changed randomly over 0–50% range about the nominal values , etc. In the current study, the upper bounds of the external disturbance and minimum estimation errors were updated online with adaptive laws without any offline tuning phase. Although increasing the effect of the external disturbance or uncertainty degraded the tracking performance, the results of Figs. ε0 was used. Another challenge facing the MIMO structure of the controlled system is the adjustment of several control parameters to achieve high tracking accuracy and robustness in the presence of external disturbance and uncertainty. To reduce the complexity of the centralized MIMO closed-loop controller with high calculation and easily select the control parameters, a decentralized form of the controller may be an alternative practical approach for the simple implementation of the controller in real DBS systems. The superiority of the decentralized structure is that the MIMO dynamical model is converted to the separated single-input single-output (SISO) subsystems with each controller acts solely on its subsystem, and the interactions between the subsystems are approximated with the adaptive fuzzy controller embedded in each subsystem. The performance of the decentralized closed-loop DBS using AFTSMC should be further investigated and constitutes future research.This paper developed a three-input (continuous DBS pulses), three-output (membrane voltage of neuron) model-based controller using robust AFTSMC to suppress the spiking pattern in a CAE dynamical model. The proposed model consists of cortical, thalamic relay, and reticular nuclei neurons. Although in the previous workIn the current study, the structure of the MIMO-controlled system was designed based on the highly nonlinear (conductance-based Hodgkin-Huxley type neuron model) CAE dynamical system, and the adaptive fuzzy controller was recruited to estimate the unknown dynamics of the plant. One important issue that should be considered in a real closed-loop DBS system is the speed of the control algorithm to meet the real-time application. Regarding the computational complexity of the proposed control algorithm, with the sampling period of 0.001 ms, when the algorithm was executed on the CPU the mean run time of the closed-loop system for the total time of the simulation (2000 ms) was about 25 min. It is worth noting that a CPU sequentially processes the data. But, in real-time applications of closed-loop DBS, the control algorithm should be implemented on an FPGA (Field-Programmable Gate Array) which is an integrated circuit with so many parallel hardware resources. Implementation of the control algorithm on the FPGA benefits from parallel data processing and this leads to a great reduction in the run time of the proposed procedure, which may be addressed in future studies.76 may be generally used to deliver the stimuli pulses. But, to satisfy neural tissue safety, the pulse trains of charge-balanced biphasic currents are delivered into the neural tissues78 and the stimulation parameters  can be adjusted individually or simultaneously with the control algorithms. Short pulse width reduces the damage risk of the neural tissue, and the increment of the pulse amplitude depends on the spatial relationship between the tip of the electrode and the target neural fibers. The high frequency of the stimulation can directly affect the average power consumption of the DBS hardware and decreases its battery life. Another problem is the spread of the electric field due to the location and geometry of the tip of the electrode during selective stimulation of the special neural tissues. The admissible amplitude or width of the stimuli pulses should be appropriately adjusted to focally stimulate the target brain nucleus with the lowest effect on the neighboring non-target tissues. In the current study, the control gain of the model in  and biphasic (with and without inter-pulse delay between symmetric and asymmetric anodic and cathodic phases) waveformsmodel in  was cons79, ECoG80, electromyogram (EMG)80 and kinematic (acceleration)81. Selection of the appropriate biological signals depends on resolution, invasiveness, and relevance to the patient's symptoms. In previous closed-loop DBS methods, the recorded biosignal time series were not directly applied as the input to the controller. In fact, the researchers processed the recorded signals and extracted different features to control the stimulator output. Synchrony of the neural oscillations48 and sub-band power of the brain signals82 are examples of the considered features in DBS which are relevant to the disease symptoms. But, in the proposed closed-loop DBS technique, the recorded membrane voltage directly drives the controller without any additional processing stage. In other words, in the proposed method, the control signal varies in a continuous manner such that the output of the plant tracks the desired membrane voltage. Since the stable intracellular recording of the action potentials requires periodic calibration of the experimental setup, the long-term clinical application of this approach is limited. To eliminate the limitation of the tracking-based controller and make it suitable for clinical applications, the time series tracking error should be substituted by an appropriate feature tracking error. A candidate feature can be the synchrony of the LFP signals of the brain nuclei. Another limitation of the proposed control scheme in designing the control input is the delay of the model states. In this paper, to estimate the nonlinear dynamics of the delayed states in the input of the fuzzy estimator it was assumed that the value of the delay is known. To hardware implementation of the proposed control algorithm in a real-time closed-loop DBS system, since the delay in the nonlinear dynamics of the system is unknown it should be approximated online using the time-delay estimation (TDE) methods84 such that the stability of the closed-loop system including TDE algorithm is guaranteed.In the closed-loop stimulation approaches, different physiological signals are used in controlling the neurostimulator output. These data sources include LFPSupplementary Information.


In the noise level and repeatability tests, the standard deviations of the RDR of two types of wheat measured by the sensor were much lower than 1.0%, indicating that the sensor could accurately collect the RDR of wheat. In the calibration test, the root mean square errors (RMSE) of protein and moisture content of wheat in the Test set were 0.4866 and 0.2161%, the mean absolute errors (MAEs) were 0.6515 and 0.3078%, respectively. The results showed that the NIR phenotypic sensor about grain quality developed in this study could be used to collect the diffuse reflectance of grains and the moisture and protein content in real-time.A near-infrared (NIR) spectrometer can perceive the change in characteristics of the grain reflectance spectrum quickly and nondestructively, which can be used to determine grain quality information. The full-band spectral information of samples of multiple physical states can be measured using existing instruments, yet it is difficult for the full-band instrument to be widely used in grain quality detection due to its high price, large size, non-portability, and inability to directly output the grain quality information. Because of the above problems, a phenotypic sensor about grain quality was developed for wheat, and four wavelengths were chosen. The interference of noise signals such as ambient light was eliminated by the phenotypic sensor using the modulated light signal and closed sample pool, the shape and size of the incident light spot of the light source were determined according to the requirement for collecting the reflectance spectrum of the grain, and the luminous units of the light source with stable light intensity and balanced luminescence were developed. Moreover, the sensor extracted the reflectance spectrum information using a weak optical signal conditioning circuit, which improved the resolution of the reflectance signal. A grain quality prediction model was created based on the actual moisture and protein content of grain obtained through Physico-chemical analyses. The calibration test showed that the R Real-time non-destructive testing of grain quality can lower the cost of field management by providing reference information for fertilization in the grain production process. It can also provide information for quality-based pricing during trade and provide data for grain classification to improve the efficiency of grain processing. However, traditional grain quality detection methods mainly rely on indoor Physico-chemical analysis. For instance, the grain moisture content was determined by the direct drying method , and the™ Lab/Pro spectrometer (570–1850 nm), Near-infrared (NIR) spectroscopy has developed rapidly as an efficient, green, and nondestructive analysis and detection method in recent years. The response characteristics of the NIR reflectance spectrum of grain are closely related to grain quality. The quality of grains can be described quantitatively using the NIR wavelength that is sensitive to the quality content . By collBased on the NIR spectroscopy technology, grain quality detection devices have been developed. Zhang et al. designed a NIR analysis system with a wavelength of 800–1,100 nm based on a charge-coupled device and fixed optical grating, and a wheat moisture and protein model was created using the PLS method. However, this system was complex and large, it could not be used for real-time on-site detection . Hidaka Studies on the analytical techniques based on NIR spectroscopy have played an important role in grain quality detection. However, current studies are still facing the following problems.Commercial instruments mostly consist of spectrometers operating in the full spectral band. They are complex, bulky, expensive to produce, difficult to adapt to different detection requirements, and do not have a grain quality detection model; thus, they are incapable of outputting grain quality information in real-time.Most instruments made in-house consist of spectrometers operating in characteristic spectral bands. The incident light spots of multiple wavelength light sources suffer from the problem of inconsistent detection regions. For grains with uneven distributions of nutritional quality, inconsistent detection regions will result in systematic errors, thereby rendering the instrument unsuitable for grain quality detection.In this study, a NIR phenotypic sensor about protein and moisture in wheat grain (PSPMWG) was developed. Compared with previous studies, this study made contributions in the following aspects.A NIR PSAGQ was developed, which had a simple structure, was lightweight and was able to detect the grain quality rapidly, nondestructively in real-time. The sensor collects the diffuse reflectance spectrum in real-time and couples the quality detection model to generate grain quality data immediately. In addition, this study expounds on the optical system design and control system development process of the phenotypic sensor in detail, which lays a good foundation for the secondary development of the phenotypic sensor and the application of online detection.A circular structure of multiple light sources was proposed, where the incident light spots of the multiple light sources were consistent, which reduced the measurement error under repeated sample loading conditions caused by the differences in incident light spots. Different incident light spots could be formed by multiple light sources on the detection plane. The overlapping circular part was intercepted to ensure the consistency of incident light spots from multiple light sources and to improve the spectral accuracy of the sensor.A detection model of wheat protein and moisture was built, which was able to output the grain quality information in real-time after obtaining the grain spectrum. The model could be directly used in multiple instruments through the self-correcting luminous state by correcting the reflectance of the sensor through the diffusion fabric and correcting the sample pool. The instrument could be used during grain production, trading, and grain processing.There are two methods for measuring the NIR: the transmission method  and the Wheat grain is the research object of this study. Its nutritional components are unevenly distributed in the grain. The PSPMWG can obtain the nutritional quality information of the wheat through the grain spectrum measured using the diffuse reflection method.The relative diffuse reflectance (RDR) is defined as:where I is the diffuse reflection light intensity of the measured sample, and Diffuse reflectance spectral signals can also be characterized by absorbance, as defined byThis work used RDR to characterize the diffuse reflectance spectrum. When measuring the diffuse reflectance spectrum, each measurement result of the sample is affected by the loading conditions. Because the internal arrangement and distribution of the samples loaded each time are different, the optical path will change, which will lead to spectral changes and poor spectral repeatability. With the design of a closed sample cell and annular light sources at multiple spectral bands, a NIR PSPMWG could reduce the influence of sample loading conditions on spectral measurement and improve spectral repeatability.™ Lab/Pro spectrometer  and a control system (including hardware circuits and software). The light-emitting unit generates incident spectra, the spectral collection unit obtains the RDR of the grain, and the hardware circuit is used to drive the light-emitting unit and the spectral collection unit. The software couples the RDR with the grain quality detection model to determine the quality content values, which are then displayed on the liquid-crystal display (LCD) with an audible output. The overall structure of the sensor is shown in via the photoelectric detector.A stable and reliable optical system is very important for the NIR PSPMWG to obtain the diffuse reflectance spectrum accurately. The light-emitting unit provides a stable and reliable incident spectrum through NIR light sources, and the spectral collection unit collects the diffuse reflectance spectrum of the grain accurately To obtain stable reflectance spectrum signals during grain quality detection, the sensor needs to have light sources with large luminous intensity and stability. Common NIR light sources include a halogen lamp and LED. The halogen lamp is bright and has a long service life, yet it is large and has high power consumption and poor stability of luminous intensity. In addition, the halogen lamp has poor timeliness because it has to be preheated for 30 min before use . The LEDThe light-emitting unit was designed based on the selected NIR LEDs, the light-emitting path of the light source is shown in According to the above design, the incident light spots of the four light sources are formed in different areas of the detection plane, leading to system errors. Thus, a circumferential layout with even distribution of four LEDs was used . The incThe light-emitting unit was simulated and analyzed by the TracePro optical simulation software to verify the consistency of the incident light spots by the four light sources. First, four light sources with an incident angle of 45° and a half-value angle of 12° were constructed, and a light beam blocking plate was placed at the appropriate location to substitute the detection plane in the actual design. The simulation results of the light-emitting system are shown in A photoelectric detector is an electronic component that converts optical signals to electric signals , which e2. The collection of diffuse reflectance spectrum by the photoelectric detector is shown in The spectral collection unit was designed based on the selected photoelectric detector. The diffuse reflectance spectrum emitted from the detection window was a circle with a diameter of 20 mm, while the photosensitive area of the selected photoelectric detector was a 3.6 × 3.6 mmTo improve the collection accuracy in the diffuse reflectance spectrum, all diffuse reflectance spectrum of grain has to be received by the photoelectric detector, that is, the circular diffuse reflectance spectrum with a diameter of 20 mm needs to be reflected on the square photosensitive area. Therefore, it is necessary to add a condenser between the detection window and the photoelectric detector. The Fresnel lens is developed based on the Fresnel theory , it is sThe calculation of the focal length of the lens is as follows:where D is the dimension of the reflection spot, d is the dimension of the photosensitive element, and H is the distance from the light spot to the lens.Based on the dimensions of the spectrum collection unit, the focal length of the Fresnel lens was 15 mm, and the diameter was 20 mm, with a thickness of 2 mm. The above parameters were substituted to the calculation formula  of FresnThe spectrum acquisition unit was simulated and analyzed by the TracePro optical simulation software to verify the light-condensing performance of the Fresnel lens used in the system. The Fresnel lens was constructed based on a theoretical design, and a light-blocking plate was placed at an appropriate location to replace the photosensitive plane in the actual design. Then, the light-condensing performance of the lens was assessed by a parallel light beam source. The simulation results of the spectrum acquisition unit are shown in The control system consisted of hardware circuitry and a software system. The hardware circuit is composed of the light source driver circuit, the spectral collection circuit, and the signal conditioning circuit. The hardware circuit is mainly used to ensure stable luminescence of the light source and accurate collection of the diffuse reflectance spectrum of grain. The software system mainly achieves the collection of the diffuse reflectance spectrum of grain, couples that data with the grain quality prediction model, and outputs the grain quality results.The luminous intensity of the light-emitting unit needs to be stable for the PSPMWG to accurately collect the diffuse reflectance spectrum. The luminous intensity of the light-emitting unit is affected by its factors  and driver factors such as drive current. The light source is driven by the optical power feedback circuit. The light source driver circuit based on optical power feedback mainly consists of four parts: optical power feedback, comparative amplification, constant current drive, and compensation filter. The circuit diagram is shown in When the luminous power of the light source under constant current reduces, the feedback voltage in the optical power feedback circuit decreases, the drive voltage in the comparative amplification part increases, the drive current in the constant current driving part increases, and the luminous power of the driving light source increases, and vice versa. In the compensation filter part, the light source drive signal is stabilized by reducing the ringing and overshoot driven by the modulation signal through the resistive–capacitive compensation filter circuit. The drive current and luminous power of the light source can be balanced through the optical power feedback to achieve stable luminous power of the light source.The accuracy of the grain diffuse reflectance spectrum obtained by the spectrum collection circuit directly affects the accuracy of grain quality data. In the spectrum collection circuit, the diffuse reflectance spectrum is first converted into light current through the photoelectric detector, which is converted into a voltage signal using the current–voltage conversion circuit, then the noise signal is filtered. Lastly, the voltage signal is amplified to the collection range of the microprocessor for further processing and output.The spectrum collection circuit mainly consists of three parts: current–voltage conversion, filtering, and amplification. The current–voltage conversion circuit is shown in The software system is composed of two modules: initialization and functional control. The initialization module is used for microprocessor initialization, pulse width modulation (PWM) signal initialization, digital-to-analog conversion initialization, keyboard input, and LCD initialization. The functional control module is used for light source time-sharing drive, diffuse reflectance spectrum collection, diffuse reflectance spectrum correction, grain quality detection, keyboard input control, LCD output, and audible output. Light source time-sharing driving means that the digital input/output ports of the microprocessor generate PWM signal and light source control signals, and then the PWM signals make the LEDs generate the modulated light signals. Diffuse reflectance spectrum collection is the collection of modulated light signals through a spectrum collection circuit. Diffuse reflectance spectrum correction is performed by correcting the luminous state of the LEDs in real-time through the correction sample cell . Grain qAs shown in Three types of tests were conducted: correction, performance, and calibration. In the correction test, multiple PSPMWGs were used to measure the relationship between the RDR and the standard diffuse reflectance of the diffusion fabric. The purpose is to ensure the universal application of the grain quality detection model in multiple PSPMWGs. Diffusion fabrics with standard diffuse reflectance of 6.5, 25, and 48% were used as the test material in the correction test. The RDR of the diffusion fabrics was measured by the PSPMWG, and the mean value of each diffusion fabric measured five times was taken as its RDR. Then, the linear relationship between the RDR and the standard diffuse reflectance of the diffusion fabric was established.In the performance test, the RDR of the same grain was measured several times to verify the stability of the NIR PSPMWG. Two wheat grain samples  were selected in the performance test. The PSPMWG was used to collect the noise level of the RDR instrument five times, and the repeatability performance of the RDR instrument was accessed under the repeated sample condition.via physicochemical analyses. Then, the relational model between RDR and physicochemical analyses values was created using the chemometrics method, i.e., the wheat grain quality detection model.In the calibration test, a relational model between the RDR and the physicochemical analyses values was built such that the moisture and protein contents could be directly obtained once the RDR was measured by the PSPMWG. Twenty-four types of wheat grain samples  were selected in the calibration test. The PSPMWG was used to collect the RDR of wheat grain; the mean value of each sample measured five times was taken as the RDR, and the moisture and protein contents of 48 samples were measured The test equipment includes PSPMWG and diffusion fabrics. A physical image of the NIR PSPMWG is shown in In the physicochemical analyses of moisture and protein contents of wheat, the conventional ISO method  was usedR2), RMSE, and MAE.There was a nonlinear relationship between the RDR of wheat grain and its physicochemical analyses values. Due to the interaction between the moisture of wheat grain and the components such as protein, the noise of the sensor and other factors can also cause nonlinearity, and the ideal model cannot be obtained by the linear correction method. The artificial neural network method has high nonlinear expression ability and is widely used to establish nonlinear NIR analysis models. In the PSPMWG, the Neural Fitting (nftool) app in MATLAB was used to establish the grain quality, detection model. The nftool is a two-layer feed-forward network with sigmoid hidden neurons and linear output neurons, that can fit multi-dimensional mapping problems arbitrarily well, neural network is shown in (1) (R2)i, n is the number of samples. With the same real value range, the closer R2 is to 1, the better the regression or prediction of the model.where (2) RMSEwhere he model .(3) MAEwhere R2 were all above 0.99. The diffuse reflectance correction equations of 780, 910, 980, and 1,050 nm were y = 0.2560 × −0.1227, y = 0.1515 × −0.0541, y = 0.1911 × −0.0845, and y = 0.196 × −0.1042.The diffuse reflectance correction of the NIR PSPMWG is shown in The noise level and repeatability of the NIR PSPMWG are shown in The relationship between the detection values obtained by the NIR PSPMWG through the quality detection model and the physicochemical values is shown in The NIR PSPMWG based on NIR spectroscopy can quickly, efficiently, and nondestructively obtain grain quality information. Currently, full-band NIR commercial instruments such as NIR Systems 5,000 spectrometer and FOSS Infratec 1,241 spectrometer have achieved good results in the application of grain quality detection . Liu et Due to the different sensitive wavelengths of different grain quality components, the PSPMWG needs to change the wavelength of its light source for specific detection objects. In addition, the shape, size, and size distribution of different varieties of wheat grains affect the scattering coefficient, which can lead to measurement error in the RDR. Further studies are needed to explore the methods that can reduce the difference of scattering coefficient and improve the accuracy of RDR and ultimately the detection accuracy of the PSPMWG.A NIR PSPMWG was developed. An optical system with a multi-source circular structure was used to avoid the difference of incident light spots of multiple light sources. The light source drive circuit based on optical power feedback and the weak optical signal conditioning spectral acquisition circuit was designed to improve the luminous stability and signal-to-noise ratio of the sensor. The standard deviation of RDR at four wavelengths was less than 0.17 and 0.51% in noise level and repeatability test, indicating that the sensor developed in this study can accurately collect RDR of wheat grain.The quality detection model of wheat grains was established. The RDR of wheat grains was collected by the phenotypic sensor, and the physicochemical values of moisture and protein were obtained by physicochemical analyses. In the Test set, the RMSEs of protein and moisture content of wheat were 0.4866 and 0.2161%, the MAEs were 0.6515 and 0.3078%, indicating that the phenotypic sensor can achieve real-time, on-site acquisition of RDR and quality information of wheat grains.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.YL: data curation, visualization, writing—original draft. YL and DL: formal analysis. JN: funding acquisition and project administration. YL, DL, and HL: investigation. JN and YL: methodology and writing—review and editing. WC and YZ: resources. YL and XJ: software and supervision. XJ, YL, and HL: validation. All authors have read and agreed to the published version of the manuscript. All authors contributed to the article and approved the submitted version.This work was supported by the National Natural Science Foundation of China [grant number 31871524]; the Primary Research & Development Plan of Jiangsu Province of China ; Six Talent Peaks Project in Jiangsu Province [grant number XYDXX-049]; and the 111 Project [grant number B16026].The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


The fault diagnosis of power transformers is a challenging problem. The massive multisource fault is heterogeneous, the type of fault is undetermined sometimes, and one device has only met a few kinds of faults in the past. We propose a fault diagnosis method based on deep neural networks and a semi-supervised transfer learning framework called adaptive reinforcement (AR) to solve the above limitations. The innovation of this framework consists of its enhancement of the consistency regularization algorithm. The experiments were conducted on real-world 110 kV power transformers’ three-phase fault grounding currents of the iron cores from various devices with four types of faults: Phases A, B, C and ABC to ground. We trained the model on the source domain and then transferred the model to the target domain, which included the unbalanced and undefined fault datasets. The results show that our proposed model reaches over  This section illustrates the research’s motivation, target and framework, followed by a relevant literature review of the recent research about the fault diagnosis of power transformers. After that, we state the research gap, our contributions and this paper’s organisation.Power transformers are important equipment in electrical power systems, and transformer failures will negatively impact whole systems. An early-stage diagnosis of transformer fault type can save the high cost of repairing and downtime. The three-phase grounding currents of the iron core generated during the turning on of transformers are important indexes to assess the states of power transformers. The value of the normal grounding current of the iron core is quite small, while the abnormal grounding current value will increase substantially. The warning threshold for the grounding current of the iron core is 100 mA for the 110 kV power transformers and the faults are caused by Phases A, B, C and ABC to ground fault. This research aims to construct a reliable model that can analyse the fault type after 10 min from fault occurrence based on the abnormal three-phase grounding current data of the iron core, and the model can be quickly transferred to different devices with quick retraining to obtain good performance.Benefiting from the evolution of machine learning, more and more supervised or unsupervised models have been adopted to pinpoint the linear or nonlinear relationships between electrical features and fault types . This paBesides that, we compose a new framework of semi-supervised transfer learning called adaptive reinforcement (AR). This technique aims to solve the problem of unbalanced and undefined fault type data in the target power transformer, which negatively affects the transferring process. The first step of AR is pretraining the source model’s classifier and then giving pseudolabels to the unlabelled data by the trained model. After that, we supplement the unbalanced fault data from the source domain based on the probability distribution. To overcome the problem of overfitting the complemented data, we replay the training experience of the main type of fault. Finally, we fine-tune the whole network’s weights with few epochs to fit the target device.The comprehensive framework of the methodology is shown in Research on fault diagnosis and the detection of power transformers focuses on different aspects and methodologies, separately. The frequency response analysis (FRA) technique is one main method to detect transformer tap changer faults. The experiment in  was cond work in  adopts tA well-trained model being adapted to similar tasks has become a popular technique with the advantages of saving time and having better model performance in the engineering area. Paper  split thThe literature review explored several novel works on fault diagnosis and transfer learning. FPA-based methods mostly focus on transformer tap changer faults by calculating the correlation coefficients. Deep-learning-based and machine-learning-based frameworks mostly utilize dissolved gas analysis (DGA) to establish fault classification. Transforming methods, including the Hilbert transform, fast Fourier transform and wavelet transform, study the information contained in the frequency to predict the occurrence of the fault. For transfer learning, fine-tuning is still the most popular method. For semi-supervised transfer learning, the novel proposed techniques are mostly based on the ideas of target domain adaption and consistency regularization.This paper proposes a deep neural network to figure out the fault types of power transformers. The architecture is formed by the FCNN, attention mechanism and BiLSTM. For the semi-supervised transfer learning task, AR enhances the domain adaption step by mixing the dynamic time wrapping algorithm. In addition, the experience replay step reinforces the experience of training the major fault.To the best of our knowledge, we are the first to combine the semi-supervised transfer learning method with the experience replay idea and utilize this method to solve electrical system problems.We combine fully convolutional neural networks (FCNN) with an attention mechanism followed by bidirectional long short-term memory (BiLSTM) networks as the diagnosis model’s architecture to classify the fault type accurately.Considering the unbalanced and undefined fault data in the target task, we propose an enhancement on the pseudolabel method based on cross-entropy and dynamic time wrapping algorithms.Our main contributions can be summarized below:The rest of the paper is organized as follows: This section specifies the characteristics of the datasets, exploration methods and preprocessing steps as the setup for the experiments.The datasets we employed for this research were the 3-phase fault grounding currents of the iron core data created by the main 110 kV power transformers owned by State Grid Shanghai Municipal Electric Power Company. We recorded the fault current after it reached the warning threshold, which was 100 mA. The source domain samples were the fault grounding current data of the iron core by one 110 kV main power transformer produced last year with four types of faults: Phase A to ground, Phase B to ground, Phase C to ground and Phase ABC to ground. The target domain samples were three different 110 kV power transformers’ fault data which contained labelled and unlabelled fault types. The value of the normal grounding current of the iron core was below 10 mA, and the abnormal current value increased substantially to about 150 mA. After the fault happened, as the abnormal current value appeared, we waited until the fault data had enough length and then input it into the model to classify the fault type.The source data were collected from one main power transformer. Three detection devices monitored Phases A, B and C’s grounding currents of the iron cores individually. The detection devices recorded the corresponding grounding current data of the iron core simultaneously every twenty seconds. The maintenance worker found out the fault’s reason and recorded it each time.We selected the fault data The source domain collected four fault types’ data which are labelled. The next sample was one time-step shift from the last sample until the faults were fixed, where the time interval was twenty seconds. For example, the first sample was The target domain’s 3-phase fault grounding currents of the iron core data were collected from three 110 kV diverse power transformers. The detecting conditions, the time-steps and the data shape were the same as those of the source domain. The values of the fault data were similar to the source domain’s data.The difference is that the fault types’ data of the target domain were combined with unlabelled and labelled types’ data. The labelled data were heavily unbalanced as well. Data cleaning is the first step in data analysis. We pruned and checked the missing data and outliers due to the error of the detection device in the samples.X = X equals the number of real values T. A key concept of time series prediction is that the data are expected to be stationary, which means the mean and variance of the data will not be varied too much with the passing of time. To improve the stationarity of the samples, we applied Lag2 differencing , an attention mechanism and bidirectional long short-term memory (BiLSTM). The whole framework is shown in The first architecture is the one-dimensional fully convolutional neural network for its robustness, as it gains good performance in multiple series tasks to extract the hidden information. The input of the networks is the three-phase fault grounding current of the iron core data in the shape of t (Relu)  as an acThe following is an attention mechanism . The locThe first layer of attention is a dense layer with the activation functions of softmax . The denA is a matrix consisting of output vectors N is the input_dimension.The feature at one time point is multi_dimensional, and the multiple features change with time, so the computed weight vector is multi_dimensional. This paper averages the weight vector to compute the attention score ctivated , where AThe last step is repermuting the weight vector-matrix back into a  shape and then multiplying the input . We add T. The computation process transfers the useful information from the cell state, drops the useless information and outputs the hidden state at each timestamp. The drop, memory and output are controlled by the last timestamp’s hidden state The next layer is bidirectional long short-term memory (BiLSTM) , which iThe joint of the forward LSTM outputs The output of the model is a probability distribution of the four fault types generated by the dense layer. The fault is classified according to the highest probability.l indicates labelled and unlabelled examples u indicates unlabelled.The whole framework of semi-supervised transfer learning, which we call adaptive reinforcement (AR), is shown in The first step of semi-supervised training is pretraining the target model. The feature extractor’s weights of the target model are initialized with For the classifier’s weights of the target model, which we denote as Compared to other pseudolabelling methods ,27, we dDTW) . Cross-eThe source model’s output, composed of -entropy  between C is the classification number, which is 4 in our task. During the experiment, we found the cross-entropy values with different fault types were very close for a few unlabelled samples. For example, one unlabelled sample’s cross-entropy values with fault Type 2 and fault Type 3 are 0.34 and 0.35, separately. It would be unscientific to assign this sample to one type directly. We set an s than ε , and we n ε  is the distance of The principle of the After pretraining and assigning the pseudolabels, we obtain a target model composed of shown in .(8)Nsw=The complete target dataset becomes odel DQN .Q, the sample of the most common fault as For each batch during the training process, we denote the initial target model as  process  as belowQ for the specific sample during the replay process. We multiply the max value of the new output vector, which is the probability of predicting the most common fault, with a gamma which we set as 0.01. Then, we add the multiplication value to the raw vector’s corresponding fault probability as the new output vector We obtain a new output vector using the trained model This presents the results of the training processes on the source target and transfer learning results of the three target power transformers.The source domain contains 7122 samples, which we shuffled randomly and split into The training history of the validation dataset is shown in For more Accuracy , Recall   are generated. The model inputs the preprocessed fault data and returns the probability distribution of the four types of faults. If the largest probability is higher than the confidence level, which we set as 0.7, the system will deliver the corresponding fault type message to the maintenance worker. On the other hand, if the four probabilities are close, the system will wait another twenty seconds until the next detection data are generated and then replicate the above steps.


Expression recognition is a very important direction for computers to understand human emotions and human-computer interaction. However, for 3D data such as video sequences, the complex structure of traditional convolutional neural networks, which stretch the input 3D data into vectors, not only leads to a dimensional explosion, but also fails to retain structural information in 3D space, simultaneously leading to an increase in computational cost and a lower accuracy rate of expression recognition. This paper proposes a video sequence face expression recognition method based on Squeeze-and-Excitation and 3DPCA Network (SE-3DPCANet). The introduction of a 3DPCA algorithm in the convolution layer directly constructs tensor convolution kernels to extract the dynamic expression features of video sequences from the spatial and temporal dimensions, without weighting the convolution kernels of adjacent frames by shared weights. Squeeze-and-Excitation Network is introduced in the feature encoding layer, to automatically learn the weights of local channel features in the tensor features, thus increasing the representation capability of the model and further improving recognition accuracy. The proposed method is validated on three video face expression datasets. Comparisons were made with other common expression recognition methods, achieving higher recognition rates while significantly reducing the time required for training. With the rapid development of the internet, the visual information that can be seen in everyday life is taking on an increasingly rich form. Compared to traditional 2D static images, 3D image sequences contain more information, especially temporal information, such as behavioral actions. Facial information is one of the most common sources of information in everyday life, and the visual information it conveys is of great importance ,2. In reTraditional research methods can be divided into two categories: feature extraction based on static images and feature extraction of video sequences. Among these, the static images-based feature extraction methods can be subdivided into holistic and local methods, as the static images of facial expressions visualize the changes in facial shape and texture produced by the movement of the muscles of the face as expressions occur. As a whole, this apparent deformation of the facial organs can have an impact on the global information of the face image, hence the emergence of facial expression recognition algorithms that consider expression features from a holistic perspective, such as principal component analysis (PCA), independent component analysis (ICA) and linear discriminant analysis (LDA) ,18,19,20In the field of image recognition, especially in the field based on deep learning, the training speed of the model and the recognition accuracy of the model are usually difficult to be achieve at the same time. Deep learning models are often complex in structure and the number of samples available for training has proliferated with digitalization; once new samples or labels are added, the entire model has to be trained all over again. Such low timeliness often does not meet the actual demand. Therefore, how to reduce the time required for model training while ensuring model accuracy as much as possible is a widespread concern among scholars in the field of deep learning today. This paper proposes a video sequence face expression recognition method based on Squeeze-and-Excitation and 3DPCA Network (SE-3DPCANet). 3DPCANet is a generalization of the lightweight convolutional neural network PCANet in 3D space. Thanks to the better feature extraction capability and lower computational complexity of PCA, 3DPCANet has a faster training speed compared to general deep networks and does not require a large number of training samples. 3DPCANet takes expression image sequences as input samples and introduces the 3DPCA algorithm in the convolution layer to directly construct tensor convolution kernels without weighting the convolution kernels of adjacent frames by shared weights, to extract dynamic expression features of video sequences from spatial and temporal dimensions. Then, to address the redundancy of local channel features, a Squeeze-and-Excitation Network (SENet) is introduced in the feature encoding layer, to automatically learn the weights of local channel features in the tensor features, thus increasing the representation capability of the model and further improving the recognition accuracy.The main contributions are as follows:(1) Since the complex information and great data volume of the facial expression data of video sequences and the extremely high training time cost using deep neural networks, this paper proposes use of the lightweight convolutional neural network 3DPCANet for feature extraction of the facial expression data of video sequences. Benefiting from the excellent feature extraction capability, low computational complexity, and simple mathematical mechanism of 3DPCANet, the facial expression features extracted by 3DPCANet greatly reduce the redundancy of the original information while retaining the key information.(2) In response to the redundancy of local channels, this paper introduces SENet in the feature encoding layer to automatically learn the weights of local channel features in tensor features. The proposed method increases the representation capability of the model and, in combination with 3DPCANet, greatly reduces the time cost of model training while ensuring the accuracy of the model as much as possible. The proposed method is validated on three datasets, compared with a variety of mainstream deep learning models.The rest of the paper is organized as follows. Nth-order tensor can be expressed as This section begins with an introduction to the representations of tensors and the associated operations. Assume that the Nth order index of the tensor. If Define the n-mode product of the tensor Nth-order tensors The inner product of two Accordingly, its Frobenius norm is defined as:Nth-order tensor Mth-order tensor Define the outer product of an That is, the order of the tensor W is the sum of the orders of the two tensors.The Kronecker product is the operation defined on the two matrices Wang et al.  establisAssuming a three-dimensional sequence Define the 3D covariance matrix of the sample:Equation (7) can then be transformed into a singular value decomposition (SVD) problem on the al space .Firstly, the third-order tensor As can be seen from n-th expansion of the kernel tensor Then, the two-dimensional matrix singular value decomposition of The attention mechanism can be seen as a generic pooling method for adaptively assigning weights to inputs and has begun to be used extensively in convolutional neural networks in recent years, providing a huge performance boost in many tasks. Convolution operations can fuse spatial and channel features, and most research has focused on optimizing models for spatial features. The channel attention mechanism, on the other hand, addresses inter-channel relationships and proposes a new Squeeze-and-Excitation Net (SENet). Each feature of the convolution output can be seen as a channel, and the core of this lies in the fact that a constant weight can be predicted for each convolution output feature channel by automatic learning .The basic structure of the SENet is shown in CC channels, we get Assuming that the convolution outputs C, dimension C, dimension That is, the feature map of channel The second step then performs the Excitation operation as follows:Z  obtained after global pooling to obtain a ReLU activation. A full concatenation of the activation results was then performed, turning the C-dimensional vector and performing a Sigmoid activation (mapping the weights to between 0 and 1), resulting in a weight matrix.Equation (11) specifically contains two fully connected layers, and the first full connection is performed on the result Finally, the weights of the learned channels are multiplied by the initial features to output the attention-weighted feature map:The attention-weighted corrected feature map allows for the retention of valuable features and the elimination of less valuable ones. SENet is generic and can therefore be added to existing network architectures, with only a small increase in computational consumption, but with a significant increase in network performance. The overall operation of the SENet is shown in In this paper, we propose a video sequence face expression recognition method based on SE-3DPCANet. As shown in N sets of facial expressions t, which is used as the sample input tensors.Suppose we take a video sequence of ith sample jth tensor sampling block of sample N sets of samples yields Let the tensor sampling block size be Let Then sample Before inputting the second convolution layer, the edges of Then the second convolutional layer will eventually output Remark. This section extracts the temporal and spatial features of the video expression sequences based on the 3DPCA and uses this to construct the corresponding 3D convolution kernels for tensor convolution operations, effectively preserving the temporal information of expression changes. However, the feature tensors contain multiple local feature channels, which have some redundancy and are not conducive to the feature representation of the model if directly binarized hash encoding is performed. Therefore, the encoding of the tensor features is improved in the next subsection.Local channel features at adjacent spatial locations in each tensor feature map output after 3DPCA convolution often have some correlation, due to the overlap of sensory fields. However, in the feature encoding process of a typical PCANet, for each feature map, only weight is usually assigned by contribution and all local features are weighted based on this weight, ignoring the relationship between local channel features and also failing to dynamically adjust the weights of each feature according to the input.Therefore, this paper addresses the redundancy of local channel features and introduces the SENet to learn adaptive weights for local channel features in each tensor feature to improve the representation capability of the network. The specific steps are as follows.First, the individual feature tensor maps are still binarized to obtain a feature tensor containing only 0 and 1 elements:ReLU activation function, and Sigmoid activation function.After compressing the feature map to obtain a global description of the features, the Excitation operation is then used to calculate the weight matrix:Performing the same operation for The final attention-weighted feature map tensor C tensor blocks and the histograms of each tensor block are counted to obtain a histogram tandem feature The samples can output a final video expression feature matrix after feature extraction using SE-3DPCANet, and then the SVM classifier is used to achieve expression recognition.Remark. This subsection uses SENet to learn adaptive weights for local feature channels on the tensor feature map after binarization, reducing feature redundancy. It also avoids the need to weigh all channels based on a single weight during hash coding, increasing the representational power of the model and contributing to subsequent classification accuracy.In this paper, three video face expression datasets, CK+ , AFEW 4 and CASMThe CK+ expression dataset consists of a total of 593 video sequences, including 327 tagged expression sequences consisting of seven expressions: anger, contempt, disgust, fear, happiness, sadness, and surprise, with the remaining sequences being neutral expressions. The expressions in each video sequence change from neutral to peak expressions, so 16 frames can be intercepted at appropriate intervals as the expression sequences for this experiment, and the sequences with less than 16 frames are copied from the last frame until 16 frames are made up. All expression frames are then aligned and cropped, with a uniform crop size of The AFEW dataset is derived from film footage and is more realistic and challenging than the laboratory’s posed expressions. The parts of the film containing more expressions were selected for the interception, and a total of 1809 video expression sequences were obtained, containing seven expression labels: anger, disgust, fear, happiness, sadness, surprise, and neutral. For each video sequence, 16 frames at regular intervals were selected as the experimental sample sequence and then aligned and cropped according to the same criteria.The CASME II micro-expressions dataset consists of 255 micro-expression video sequences from 26 filmmakers, containing a total of five basic expressions: boredom, happiness, surprise, sadness, and fear, as well as other expressions. Each video sequence changes from a neutral expression to a peak expression and finally back to a neutral expression. Therefore, for each video sequence, 16 frames are intercepted at regular intervals from the beginning of the neutral expression to the end of the peak, and the interval is used as a sample of the expression sequence. The same alignment and cropping operations are then performed with the CK+ dataset.This subsection conducts experiments on the selection of parameters for the proposed SE-3DPCANet model. The optimal parameter values for the network can be selected by a combinatorial comparison. Uniformly set the number of filters to The SE-3DPCANet recognition rate curves for different parameters are shown in This subsection tests the performance of SE-3DPCANet on three datasets: CK+, AFEW and CASME II, respectively. The confusion matrix for SE-3DPCANet on the CK+ dataset was first calculated, as shown in According to The SE-3DPCANet algorithm proposed in this paper is then compared with existing expression recognition algorithms 3D-CNN, 3D Inception-ResNet, Spatio-temporal manifold, as well as PCANet, KPCANet-LDA, 3D-PCANet. A comparison of the recognition performance of each algorithm on the CK+ dataset is shown in The confusion matrix for SE-3DPCANet on the AFEW dataset was calculated using the same method as shown in The SE-3DPCANet algorithm was then compared with 3D-CNN, VGG-LSTM, VGG-LSTM-Attention, 3DCNN-Attention, PCANet, and 3D-PCANet on the AFEW dataset. A comparison of the performance of the various algorithms on the AFEW dataset is shown in Comparing the recognition rates and training times of the algorithms in The final calculation of the confusion matrix for SE-3DPCANet on the CASME II dataset is shown in The SE-3DPCANet algorithm was then compared with the common expression recognition methods LBP-TOP, EVM+HIGO, CNN-LSTM, PCANet, and 3D-PCANet on the CASME II dataset. A comparison of the performance of each algorithm is shown in In this paper, we propose a video sequence face expression recognition method based on Squeeze-and-Excitation and 3DPCA Network. The model input samples are video expression sequences, and a tensor convolution kernel is constructed in the convolution layer, using 3DPCA to extract the dynamic expression features of the 3D video sequences; then, for the redundancy of local channel features, the channel attention mechanism module SENet is introduced in the feature encoding layer, to automatically learn the weights of local features in the tensor features to further improve the recognition accuracy. Experimental validation was conducted on three datasets: CK+, AFEW, and CASME II, respectively. The recognition accuracy of the model was evaluated by calculating the confusion matrix for each type of expression and comparison with other common expression recognition algorithms for analysis. Experimental results show that the SE-3DPCANet algorithm proposed in this paper can effectively recognize various types of expressions and achieve close to or even higher correct recognition rates compared to most expression recognition methods. The proposed method has great advantages in terms of training time, and the overall performance of the model is superior.We list the benefits and drawbacks as follows:Benefits:(1) The facial expression features extracted by 3DPCANet greatly reduce the redundancy of the original information while retaining the key information.(2) SENet increases the representation capability of the model and, in combination with 3DPCANet, greatly reduces the time cost of model training while ensuring the accuracy of the model as much as possible.Drawbacks:(1) While the proposed method substantially reduces the time required for training, the model accuracy is also slightly reduced compared to deep neural networks. How to maintain or even surpass the accuracy of the original model while reducing the training time of the model, and significantly improve the comprehensive performance of the model, is a future research direction.(2) How to apply the proposed method to micro-expression recognition and extract the small difference in features of human facial expression is a future research direction.


Acne detection, utilizing prior knowledge to diagnose acne severity, number or position through facial images, plays a very important role in medical diagnoses and treatment for patients with skin problems. Recently, deep learning algorithms were introduced in acne detection to improve detection precision. However, it remains challenging to diagnose acne based on the facial images of patients due to the complex context and special application scenarios. Here, we provide an ensemble neural network composed of two modules: (1) a classification module aiming to calculate the acne severity and number; (2) a localization module aiming to calculate the detection boxes. This ensemble model could precisely predict the acne severity, number, and position simultaneously, and could be an effective tool to help the patient self-test and assist the doctor in the diagnosis. Computer vision ,2 is a sConsidering their good capabilities of extracting information from pictures, the technologies of visual recognition have also been used in healthcare to help doctors diagnose diseases, especially skin diseases whose visual representations are easier to spot. Since skin health can be easily affected by the living environment and lifestyle, people who live with unhealthy habits  are more prone to skin problems. Note that although the probability of skin diseases is not related to regions and ages, skin diseases ,41,42,43This paper investigates how to use deep learning to diagnose facial acne, a common skin disease. The occurrence of acne is closely related to excessive sebum secretion, blockage of the sebaceous duct, bacterial infection and inflammatory reaction. Since there are many types of facial acne, it is quite challenging to design an expert system to diagnose all these types of facial acne. As a preliminary work on computer-aided diagnoses of facial acne, we aim to evaluate the severity and locate the acne according to the facial images. In this paper, we propose an ensemble model to assess the acne severity, numbers and positions of the facial images simultaneously in the inference. Compared with the previous research regarding acne detection through neural networks: (1) we improve the prediction accuracy in the number and severity of face acne by dataset reclassification and random sampling; (2) we introduce a localization module to predict the location of facial acne. Guided by the extracted features in the classification modules, the model here could precisely calculate the acne position, while previously reported models can hardly predict the acne severity, number and location simultaneously.In this section, we will introduce several representative studies about the diagnoses of facial acne through deep learning.Acne grading ,48,49,50Note that the above models need complex image preprocessing, including dividing the whole face region into several regular subregions according to the features in the images. Researchers also use the whole image as the input to simplify the evaluation process. In 2019, Lim et al. developed an automatic system to calculate the Investigator’s Global Assessment  scale , a crite lesions . Specifi lesions , and tak lesions  Differen lesions . They utTo locate acne in facial images to assist the doctor in diagnosis, Rashataprucksa et al. utilized Faster-RCNN and R-FCN to train an acne detection model . PrecisiInspired by the previous work on acne grading and detection, we propose an ensemble network  to assesACNE04  is a pubN) in each category. Then, we randomly choose N images in the categories with a large sample size , and duplicate the images in the categories with a small sample size . It is worth mentioning that resize operation and normalization are applied to the input images to meet the requirement of the network input.Note that predicting the acne number in each image is one of the three tasks  in the ensemble model. Thus, smoothing the sample distributions under different acne numbers is very important to the model training. Specifically, we fix the image number N denotes the total image numbers and M denotes the total class numbers . Coefficient i belongs to category j, respectively. Typically, Considering that the main task here is to calculate the acne severity and number, we adopt a cross-entropy loss function, which is very useful when training a classification problem with several classes. The loss function can be written as:(2)i does not belong to categories j, Although the number of images in different severity classes is similar, the sample size varies widely in the categories with different acne numbers. Focal loss , a loss (3)i does not belong to category j, probability distribution Another common strategy is to transform this classification problem into a regression problem. Inspired by label distribution learning ,60,61,62Compared with the classical task of image detection, acne detection in the facial images is a tough detection task, where feature boundaries between different categories are unclear. Specifically, for two images with adjacent acne numbers, they show similar acne characteristics, though they have different facial appearances. These acne characteristics are very small compared with face contours, and can easily be overwhelmed by those large features. It is quite difficult to localize small features with high similarity in different backgrounds. In this paper, we adopt YOLOv5 as the backbone of the localization module. The Focus structure in the backbone would improve the receptive fields and ensure no missing context. The CSP structure  could deal with the problem of gradient vanishing during extracting the deep features in the facial images. The SPP  structures could improve the capability of detecting tiny objects.The operation process of the localization module is demonstrated in the right panel of On the contrary, we add two full connection  layers at the end of YOLOv5 to calculate the acne severity and number. The architecture is similar to the Linear 1 and 2 blocks in the classification module. Kullback–Leibler divergence loss is adopted to optimize the two blocks. Here, there are two different strategies to train the YOLOv5 and the subsequent classification blocks: (1) Train the YOLOv5 first, and then train the two classification blocks. The output of the trained YOLOv5 is utilized as the input of the two classification blocks. (2) Train the YOLOv5 and two classification blocks simultaneously. The loss of the two classification blocks is added to the loss of YOLOv5, and the total loss function can be written as:In this section, we first introduce the training parameters and evaluation metrics of the two modules, then detail the inference performance of the classification and localization module discussed in We train the two neural networks on a single NVIDIA Tesla P100 based on the PyTorch framework. For the classification module, we choose Stochastic Gradient Descent (SGD) with the mini-batch of 32 as the model optimizer. The initial learning rate is set to 0.001 and reduced to half every 30 epochs until it reaches 120 epochs. The momentum and weight decay are 0.9 and j. The precisions on predicting acne severity and number are increased by about As discussed in We also compare our methods with previously reported work  the classification module used to calculate the acne severity and number and provide guidance for the inference of the localization module; (2) the localization module used to calculate the detection boxes under the assistance of the classification module. This is the first time that the acne severity, number and position are simultaneously predicted through deep learning, and the prediction results show good agreement with the true results. Furthermore, considering that the acne in the different body parts (such as the face and back) usually shows similar geometrical configurations , the proposed model can be further applied in detecting back acne, chest acne, etc. This method could assist patients with self-testing by taking a selfie according to the Hayashi grading criterion, and help doctors diagnose acne problems. This ensemble model will show significant applications in medical engineering.