The human auditory system employs a number of principles to facilitate the selection of perceptually separated streams from a complex sound mixture. The brain leverages multi-scale redundant representations of the input and uses memory (or priors) to guide the selection of a target sound from the input mixture. Moreover, feedback mechanisms refine the memory constructs resulting in further improvement of selectivity of a particular sound object amidst dynamic backgrounds. The present study proposes a unified end-to-end computational framework that mimics these principles for sound source separation applied to both speech and music mixtures. While the problems of speech enhancement and music separation have often been tackled separately due to constraints and specificities of each signal domain, the current work posits that common principles for sound source separation are domain-agnostic. In the proposed scheme, parallel and hierarchical convolutional paths map input mixtures onto redundant but distributed higher-dimensional subspaces and utilize the concept of temporal coherence to gate the selection of embeddings belonging to a target stream abstracted in memory. These explicit memories are further refined through self-feedback from incoming observations in order to improve the system’s selectivity when faced with unknown backgrounds. The model yields stable outcomes of source separation for both speech and music mixtures and demonstrates benefits of explicit memory as a powerful representation of priors that guide information selection from complex inputs. The human brain solves complex auditory tasks such as having a conversation in a busy cafe or picking the melodic lines of a particular instrument in an ensemble orchestra. While seemingly effortless, these tasks are a real feat given that the brain only has access to a low dimensional pressure waveform of the mixture as the primary signal and uses a common front-end pipeline to process incoming signals, regardless of their complexity . This soNeuroscience research has shed light on some of the mechanisms and neuronal architectures that facilitate adaptive listening –9. A serInspired by these principles, numerous efforts in machine hearing have leveraged these principles to process audio signals. Hierarchical and multi-resolution schemes have been widely effective in providing rich and redundant mappings of sound inputs, particularly for the task of source separation. Grais and colleagues proposed a multi-resolution architecture using convolutional filters of varying sizes to capture different parts of the spectrotemporal modulation space for source separation , 25. HieBeyond multi-resolution architectures, attention has been widely used as a mechanism that further guides processing in deep networks by incorporating weighting of local and global contexts, as shown by recent performance leaps of models such as transformers and conformers . ThroughMulti-view feature extraction: Multiple parameterized independent streams trained in parallel to capture information from different vantage points [e points , 16. In Object memory formation: An explicit memory of different auditory objects is learned to represent patterns of a particular object captured across multiple views [le views , 31. TheSelf-attentional feedback: During inference, self-attentional feedback allows memory representations in each stream to refine themselves by using information from a different vantage point in the network, in order to enhance perceptual fidelity when faced with changing backgrounds [kgrounds .Hierarchy: The basic architecture can be repeated across different hierarchical levels. Self-feedback not only modulates parallel streams within a layer but trickles down to earlier levels in the hierarchy.In the present study, we propose a framework that achieves a dual objective in the context of audio source separation: (i) a universal, modular framework that operates on both speech and music signals by forming explicit object memories and (ii) an adaptive system that selectively re-tunes memory to adapt to changes in the soundscape. The proposed system represented in Fig. We evaluate this model for two different tasks, without any adjustment to design elements or system architecture. In other words, the same model architecture is trained to attend to target speech in presence of various noise distortions (speech enhancement) and trained to attend to a particular instrument in presence of different instrumentals or vocals (music source separation). Both models differ in the number of possible targets, i.e., 2 for speech enhancement (speech and noise) and 4 for music separation . It is important to note that the proposed scheme frames the source separation problem as one where the system is attending to one auditory target (based on priors in the memory network) and ignores all others. As such, the network only outputs the foreground signal desired to be listened to (as indicated by the user), rather than multiple signals in an input mixture.The analysis to follow evaluates the effectiveness of the above mentioned mechanisms in an agnostic source separation task without exclusively targeting computational cost or high-performance computing to outperform state-of-the-art systems. In Section O denotes the maximum number of objects that can be present in the system and is pre-defined for the task. only i.e., Expanding upon the goals described in Fig. G operating at end of each stream, and (4) self-attentional feedback to re-tune memories during inference. The following subsections describe each component in detail.The proposed model has four key components: (1) different levels of the hierarchy, each composed of parallel processing streams; (2) a stream integrator at each level of the hierarchy that renders a unified view across streams; (3) memory informed attentional gating As described previously, the network consists of parallel streams, each trained independently. Let the G that is used to attend to the object of interest and is described later in this section.Each stream consists of a two-dimensional convolution node as the basic computational unit, followed by a max pooling operation for some paths. As illustrated in Fig.entclass1pt{minimaIndividual streams capture the acoustic scene and its constituent objects at differing levels of resolution and abstraction depending on the parameters of the stream and its position in the hierarchy. We hypothesize that these streams capture the acoustic scene from different vantage points, and integrating information across streams in different hierarchical levels should lead to better target separation. Effectively, stream integrators at each level act as read-outs of the auditory objects from a particular level and have similar architectures as each of the parallel streams. They consist of convolutional nodes and DCS, albeit deeper in terms of network complexity. The input to stream integrators is the concatenated embeddings of streams from the corresponding level of hierarchy along with indicator variables The attentional gating G is inspired by the principle of temporal coherence which states that when attention is directed towards a characteristic feature of the target object, this characteristic feature acts as an anchor, and all the features that are coherent with temporal activations of this anchor become bound together to form a common auditory object , 33. In H signify the embeddings of the mixture in a 3-D space with dimensions p, q, and r represent frequency channels, number of time frames, and number of hidden units respectively. For simplicity, assume we want to segregate the audio into 2 sound objects, object A and object B. Let First, the memory vector Next, the weighted activation pattern The anchored memory is then matrix multiplied with the memory vector to mimic the object memory’s pattern along the time dimension. Resultant matrix Let This process is similar to the use of a dictionary basis to segregate sources within the non-negative matrix factorization paradigm . The objIn each of the independent streams and stream integrators, source separation or enhancement is performed through the penultimate block in the stream, indicated as G. The gating block takes the output of the previous block and indicator as the input and returns the embeddings activated in coherence to the pattern learned in the memory. The local attention gating block in each of the parallel streams and stream integrators is followed by the read-out layer, a The network proposed thus far consists of multiple streams across different levels of hierarchy with stream integrator at each level integrating information across the streams. By design, the network strives to represent distinct views of information across its different paths as information flows to integrator streams and higher levels. Each stream contains its own memory of objects used to separate sources. Deployment of top-down feedback is used to refine the memories of lower layers based on output from upper layers. This top-down process focuses on only re-tuning the streams’ object memories in the attentional gating blocks of individual streams by minimizing the objective entclass1pt{minimaIn this section, we describe the datasets, network configuration, training, and re-tuning strategy in detail for both speech and music separation. The network configuration, training, and fine-tuning strategies are similar for both tasks, hence it is described together for both tasks in the subsection below.B denotes batch size and O represents the number of objects. STFT is calculated using a window size of 2048 and a hop size of 512 with a hamming window. The number of time frames in the input is fixed to 64 and with zero padding where necessary. The indicator is one hot encoded vector representing which memory to select for further processing  of the desired (attended) object; we employ the To train the parallel streams, let norm Eq. , while tD is the L1 distance between The re-tuning procedure uses the loss function presented in Eq.mixture, drums, bass, vocals, and other instruments. For each file, the mixture corresponds to the sum of all the signals. All tracks are downsampled to 16 kHz before calculating the STFT. MSS focuses on separating the mixture into 4 auditory objects: bass, drums, vocals, and other instruments. The input to the network is the magnitude spectrogram extracted from a music mixture, along with an indicator suggesting which of these 4 objects/sources is of interest. The model predicts the magnitude spectrogram of the object for which the indicator variables is equal to one. The output of the network is compared to the ground truth track and a measure of signal-to-distortion ratio (SDR) is used to evaluate the separation performance.For this task, the MUSDB18 dataset  is utiliBaseline systems: The performance of the proposed system is compared to a diverse number of other models with comparable setup. Many state of the art systems incorporate specifics about music profiles and extensively optimize to the particular domain. For example, [example, , 39 propexample, , a data-example, , and a rexample,  trained example, .To train the speech enhancement model, we use training data from a noisy speech dataset consisting of clean speech data from the Voicebank corpus . These sIn order to address different limitations often associated with performance measure, we employ 3 different metrics in order to evaluate model outcomes from different angles.Baseline systems: The performance of the system for speech enhancement was compared to 3 other baseline neural network architectures evaluated using the same train/test data: (i) a baseline CNN network with the same configuration as that of the proposed system with 5 layers, each consisting 1024 hidden units of size posed in , with thposed in .Ablation evaluations: By design, the model provides multiple outputs at different points in the network. This structure allows us to evaluate the contribution of different components of the model. In addition, we also perform an ablation study where the contribution of different streams is nullified by setting the embeddings for that stream across all layers to zero. The rest of the model remains unchanged hence allowing us to assess the degree of damage such zeroing has on the final outcome of the model at the output of the stream integrator in layer 2. This analysis is performed on the speech task as example of expected effects of model ablation on overall performance.The evaluation of the proposed system is presented in Table track is  which usTo further understand the role of presented self-feedback mechanisms in refining the selection process, we investigate two spectrogram snippets for the drums and vocals Fig. . Rows 1 Network analysis: Given the modular nature of the network, it affords a refined analysis akin to an ablation study which allows us to explore the contributions and complementary roles of different components. Median SDR [Among independent streams, faster streams  of an ensemble of 100 randomly sampled speech utterances (from the TIMIT database) processed through the different streams in the model –57. The bic rate .Fig. 6NoAblation analysis: To better assess the complementary role of different processing streams, we perform an explicit ablation analysis by nullifying specific streams in the network and evaluating the impact on the final level 2 integrator output. In this evaluation, specific streams  in both layers are set to zero while the rest of the model is evaluated. The effect of leaving one stream out on the system performance as measured by the output of layer 2 is displayed in Table 1, 2, or  in both 1, 2, or  in both This study is inspired by the brain’s ability to pick out a target sound amidst other sources that may mask, interfere, or distort the sound of interest. Proposed framework attempts to mimic the following characteristics: (i) distributed processing to facilitate sound segregation, (ii) object memory representing characteristics of sounds across the distributed system, and (iii) selective re-tuning of the target memories to adapt to a particular melody or speaker. Moreover, the proposed system does not incorporate any specific higher-level knowledge or constraints based on the structure of the signal of interest . Instead, it is primarily a signal-driven analysis that explores the discriminability of target signals from background distractors based on the distinction between the signal characteristics of classes. Music separation has often benefited from the inclusion of musical syntax and semantic models that incorporate semantic and contextual information , 61. SpeDistributed multi-scale processing: The auditory system is viewed as a multi-scale transformation, wherein the spectral and temporal dynamics of an input waveform are extracted across a network of cortical units. Neurons in the auditory cortex are sensitive to spectral energy, spectral modulations, and temporal modulations which capture the rate of change of energy along the temporal axis (rates) in addition to the joint changes in dynamics along time and frequency [requency . Studiesrequency , 64, 65.requency , 66–68. requency , 16.To achieve this rich and multiplexed representation, the proposed system utilizes parallel pathways that are trained independently. Each path is configured to best represent a certain region of the modulation space. As noted in a recent review , while tObject memory and temporal coherence: The role of the network of memories [memories , 75 is tmemories , 76, 77.memories . In EMMAmemories , 19.Naturally, the current model does not go beyond two layers of the hierarchy and falls far short of the specialized transformations of memory in the brain that span far more than the sensory space and encompass more cognitive regions of the brain . One of Selective retuning: One of the remarkable capacities of the human brain is its ability to adapt to unknown conditions [nditions , 79. It nditions –82. WhilThe adjustments of the priors cannot happen in the design phase of the system, as they have to reflect a specific unknown utterance, melody, or even noise condition. Hence, it is important to develop models that learn continuously. In the current study, we use feedback from stream integrator as guides for this adjustment process; effectively using the model’s output and its posteriors to re-tune its priors. This mechanism can be thought of as a self-correction mechanism that ultimately aims to reinforce the representation of memory with the peculiarities of a specific observation. This approach is also substantiated by a number of findings in the auditory system whereby selective attention is shown to improve the precision of a stimulus already represented in memory . ImposinOverall, this work explores the interaction between feedforward mapping principles and selective modulations (via memories and feedback) to facilitate the segregation of sounds of interest in a complex mixture. Attention in its perceptual interpretation is at the center of this selective adaptation by engaging memories of known objects and adjusting prior knowledge to modulate how incoming sounds are processed. The proposed model is our first attempt at mimicking the auditory system in an end-to-end fashion in a very simple and intuitive setting based on neuroscience studies related to the brain and hearing.


Energy harvesting effectively powers micro-sensors and wireless applications. However, higher frequency oscillations do not overlap with ambient vibrations, and low power can be harvested. This paper utilizes vibro-impact triboelectric energy harvesting for frequency up-conversion. Two magnetically coupled cantilever beams with low and high natural frequencies are used. The two beams have identical tip magnets at the same polarity. A triboelectric energy harvester is integrated with the high-frequency beam to generate an electrical signal via contact-separation impact motion between the triboelectric layers. An electrical signal is generated at the low-frequency beam range achieving frequency up-converter. The two degrees of freedom (2DOF) lumped-parameter model system is used to investigate the system’s dynamic behavior and the corresponding voltage signal. The static analysis of the system revealed a threshold distance of 15 mm that divides the system into monostable and bistable regimes. In the monostable and bistable regimes, softening and hardening behaviors were observed at low frequencies. Additionally, the threshold voltage generated was increased by 1117% in comparison with the monostable regime. The simulation findings were experimentally validated. The study demonstrates the potential of using triboelectric energy harvesting in frequency up-converting applications. Modern science and technology have advanced quickly, leading to a substantial rise in ultra-low-powered electronics across various industries. This increases public knowledge of ambient energies and creates a sizable market for them as potential sources to replace chemical batteries in these devices . Using rThere are a variety of mechanisms that have been utilized to harvest vibrational energy for electricity generation, including electrostatic transduction mechanisms ,13, piezEven though the harvested energy from the ambient vibrations appears to be high with high-efficiency conversion devices, some devices’ resonant frequencies do not match the frequency of the ambient vibrations or the bandwidth is limited to a specific range. As a result, if the device’s frequency deviates slightly from the harvester’s resonant frequency, the harvested power will drop significantly . AdditioThis paper proposed a frequency up-converter to harvest low-frequency vibrations utilizing a triboelectric generator and magnetic coupling. The frequency-up converter mainly consists of two cantilever beams coupled with two identical magnets attached to the tip of the beams. The tip magnets are attached facing each other at the same polarity to provide a nonlinear repulsive magnetic force responsible for converting low-frequency vibrations into high-frequency oscillations. A two-degree of freedom (2DOF) lumped parameter model has been used to investigate the beam’s dynamic behavior and the produced voltage signal. The rest of this article describes the device’s setup and demonstrates how the frequency-up-converter energy harvester operates. In addition, each beam’s static response and the frequency variations caused by the magnet separation distance and the generation of voltage and power at different values of the magnet separation distance are investigated. Experiments are also carried out to validate the simulation results derived from the theoretical model. In the end, the optimum operation region is determined.d. The whole setup is attached to a holder and installed in an electrodynamic shaker to provide an excitation source to the system. The triboelectric generator mainly consists of three layers: a lower fixed electrode made of aluminum with attached Polydimethylsiloxane (PDMS) insulator layer and an upper aluminum electrode attached to the bottom surface of the HFB’s tip magnet. The upper electrode and the insulator layer are separated by an initial gap The 2DOF triboelectric energy harvester consists of two cantilever beams, as shown in nduction .X is the distance between the centers of the two magnets  system, shown in  follows :(3)m1z¨E1I1L13) , while kE2I2L23) . In addi0.375mb) , where mIt is worth mentioning that under the effect of the tip mass and base excitations enough to achieve the impact scenario as shown in the operation of the harvester, To experimentally validate the numerical results from the theoretical model, we use the setup shown in g) to include the weight effect. The weight affects only the static response, while it does not affect the dynamic response other than introducing a new equilibrium position [Since the repulsive magnetic force affects both beams statically, it is essential to investigate the static response of both beams as a function of the magnet spacing distance. Therefore, the static analysis of both beams can be formulated by setting all-time derivatives in Equation  to zero position . TherefoAt large distances between the two magnets, the magnetic force is negligible. Therefore, each beam only undergoes a static equilibrium deflection due to the tip weight effect, which is also still considered very small. However, by lowering the separation distance between the two magnets to the threshold distance, both beams start to deflect from their previous equilibrium points to settle to new equilibrium points due to the effect of the magnetic force induced by lowering magnet spacing. By varying the distance between the two magnets, the corresponding static equilibrium points can be extracted numerically by solving for the roots of Equation  and expeIn this section, we will discuss how the magnetic force affects the natural frequency fluctuation of the nonlinear harvester. Toward this, the total deflections of the beams will be based on the static and dynamics deflections as follows:Then, the magnetic force is:Using Taylor’s series around zero dynamic deflection ic force . Then, bAccordingly, the nonlinear natural frequencies of the LFB and HFB can be calculated from the following formulas:d for the LFB and HFB and then plotted as shown in The variation of the nonlinear natural frequency with varying the magnets separation distance for both beams was extracted numerically using Equation  and expeEquation  will be By removing the influence of the magnetic force α1 accordNext, the separation distance decreased to 20 mm, and the frequency–voltage curves from both the experiment and simulation are in good agreement as shown in Now, the separation distance is reduced to 17 mm, which is still in the transition regime but from the monostable side. The experiment and simulation’s frequency voltage and power curves are shown in By lowering the separation distance to reach the threshold at Experimentally, at lower separation distances below 14.5 mm, the magnetic force becomes very strong and causes the beam to stick to the fixed electrode, which could lead to failure in the system and no electrical signal generated. Therefore, the dynamic behavior of the harvester for lower distances is extracted theoretically since we already validated our theoretical model in all the previous cases. In light of this, the distance decreased to reach 12 mm, and the results of the frequency voltage and power curves of the HFB with the zoomed response in the LFB range are shown in To summarize the results discussed above and to specify the optimal operating regime, the output voltage and power at the LFB range were calculated and plotted versus the magnets’ separation distance at different excitation levels, as shown in We utilized vibro-impact triboelectric transducers with magnetic nonlinearity to create a nonlinear frequency up-converter under harmonic excitations. This study focuses on combining magnetic nonlinearity with the inherent phenomenon of vibro-impact in triboelectric energy harvesters to transfer the energy between low and high oscillations. The combination of magnetic nonlinearity and vibro-impact is a novel strategy for triboelectric energy harvesters’ applications. Moreover, the addition of magnetic nonlinearity makes the harvester variable frequency energy harvester, where the operating frequency can be controlled by controlling the distance between the two magnets to target multiple applications at different frequency ranges.In summary, a frequency up-converter using a vibro-impact triboelectric energy harvester efficiently converts low-frequency vibrations to a higher frequency is demonstrated. The system comprises low-frequency and high-frequency beams with two identical attached tip magnets. The magnetic coupling transfers the energy between the high-frequency and low-frequency ranges and generates a voltage signal at the LFB even though the generator was attached to the HFB, indicating a frequency up-converter. The structure was tested experimentally and validated with the simulated results of the theoretical model with good agreement. Different nonlinear behaviors of softening and hardening have been achieved by controlling the distance between the two magnets. Furthermore, a significant increment in the output voltage signal was achieved by lowering the distance between the two magnets. Moreover, the transition region was the optimum region to obtain the maximum voltage. Triboelectricity shows high efficiency in frequency up-conversion applications.


There are many rice diseases, which have very serious negative effects on rice growth and final yield. It is very important to identify the categories of rice diseases and control them. In the past, the identification of rice disease types was completely dependent on manual work, which required a high level of human experience. But the method often could not achieve the desired effect, and was difficult to popularize on a large scale. Convolutional neural networks are good at extracting localized features from input data, converting low-level shape and texture features into high-level semantic features. Models trained by convolutional neural network technology based on existing data can extract common features of data and make the framework have generalization ability. Applying ensemble learning or transfer learning techniques to convolutional neural network can further improve the performance of the model. In recent years, convolutional neural network technology has been applied to the automatic recognition of rice diseases, which reduces the manpower burden and ensures the accuracy of recognition. In this paper, the applications of convolutional neural network technology in rice disease recognition are summarized, and the fruitful achievements in rice disease recognition accuracy, speed, and mobile device deployment are described. This paper also elaborates on the lightweighting of convolutional neural networks for real-time applications as well as mobile deployments, and the various improvements in the dataset and model structure to enhance the model recognition performance. HoweverThe traditional rice disease recognition generally relies on the experience accumulation of farmers in the actual production process . This meWith the enhancement of computing power of computer hardware and the explosive growth of data, deep learning technology has achieved very good results and been broadly applied in many areas such as speech recognition, image processing, and natural language processing . The conIn terms of rice disease recognition, convolutional neural networks have also obtained very good results and been widely utilized  (FigureTheory of convolutional neural networks is explained detailed, especially various measures that could reduce the complexity of convolutional neural networks.Methods that can improve the accuracy of rice disease identification based on convolutional neural networks are summarized.Existing research results for speeding up convolutional neural networks to identify rice diseases are summarized.The deployment of convolutional neural networks to mobile devices for real-time rice disease identification is summarized.The limitations of rice disease recognition based on convolutional neural networks are given and the future research priorities are envisioned.The structure of this paper is organized as follows: the first part is the introduction; the second part discusses the basic principles and lightweight measures of convolutional neural networks; the third part summarizes the method of convolutional neural network to improve the recognition accuracy in the process of rice disease recognition; in the fourth part, the measures to deploy convolutional neural networks on mobile devices after lightweight revision in the process of rice disease recognition are discussed; the fifth part summarizes the approaches to optimize the recognition speed; the sixth part lists limitations of convolutional neural networks applied in rice disease identification and future research directions. Finally, a summary of the paper is presented.2The convolutional neural network is one of the classical ways of deep learning with multi-layers structure and convolution computation . It can 2.1The structure of a convolutional neural network can be split into the convolution layer, pooling layer, and fully connected layer according to different calculation ways  Figure.Convolution layer. The convolution layer collects local attributes of the training data by convolution operations . The inpPooling layer. Similar to the principle of convolution operations, pooling is realized by moving the sliding window on the feature graph . AccordiFully connected layer. The fully connected layer is positioned in the last part of the convolutional neural network. The fully connected mode is adopted to map the two-dimensional feature map into a one-dimensional vector, and finally map to the sample space according to different tasks. The preceding convolution layer and pooling layer map the input data to the feature representation space, and then the fully connected layer maps the feature representation space to the sample’s label space to achieve the final classification or regression task .2.2Traditional convolutional neural networks have the problem of too many parameters. If the model is improved to obtain better classification performance, a lot of sample sets are needed. However, in practical engineering applications, it is hard to get a lot of sample sets to promote the classification precision. There is an increasing demand for convolutional neural network deployments on mobile terminals, and in-depth lightweight research on convolutional neural networks is required. Under the premise of not reducing model performance, architecture size and the computational amount should be reduced as far as possible to get a balance between performance and overhead . At presThe commonly model compression methods are network pruning, parameter quantization, low-rank approximation, and knowledge distillation.Network pruning. Pruning, as a classical technique in the field of model compression, has been widely used in the post-processing of various algorithms. Network pruning is an important technique which could reduce network complexity and prevent network overfitting . It is wParameter quantization. Quantization is operations of replacing all the original parameters with part parameters, which greatly reduces the storage overhead. Parameter quantization, as a common back-end compression technique, can obtain a large reduction in model volume but little performance loss . The demLow-rank approximation. It decomposes the huge dense weight matrix into several small-scale matrices, the original weight matrix can be approximately reconstructed. The operation achieves the purpose of reducing storage and calculation . The basKnowledge distillation. It is a transfer learning method, and its target is the convert of the knowledge learned from the complex network framework to the compact small model over certain approaches so that a tiny scheme can also obtain a similar capability as the complex model  Figure.It is an effective method to adjust standard convolution, decrease convolution parameters and speed up convolution operation. Although the model compression takes a good part in the process of convolutional neural network lightweight, its process is too complex, and it usually requires detailed repeated training to achieve a similar performance to the original model. Therefore, some researchers begin to directly design lightweight convolutional neural network frameworks to control the number of parameters and computation by group convolution, depth-wise separable convolution, depth-wise convolution, and pointwise convolution .Group convolution. It divides the dimension of feature channels into several equal parts, then convolves them separately, and piles up the results. The idea of group convolution is widely used in network design. Besides reducing the number of parameters, it can also be regarded as a structured sparse approach, equivalent to a regularization manner. As the number of filter groups raises, the model parameters decrease, and the framework becomes more efficient. Since the convolution is partitioned into multiple paths and each path can be handled separately through a respective GPU, the architecture can be trained on several GPUs in parallel, and the training speed of the mode is greatly accelerated .Depth-wise separable convolution. The merit of depth-wise separable convolution is that the more attributes that need to be extracted, the more parameters can be saved, reducing the amount of computation. Deep-wise separable convolution is actually a kind of decomposable convolution operation, which involves spatial dimension, but also deals with depth dimension  (FigureDepth-wise convolution. It is a kind of packet convolution in which the number of packets is equal to the amount of feature channels, and the convolution kernel is one-to-one corresponding to the channels. Therefore, the depth of the output feature map is the same as that of the input. For the multi-channel characteristic graphs from the previous layer, all of them are first split into feature graphs of a single channel, and single-channel convolution is performed on them respectively and then stacked together. Different from the standard convolution activities, deep-wise convolution splits the convolution kernel into several channels and carries out convolution operations on each channel without changing the depth of the input property image. In this way, the output attribute graph with the same amount of channels as the input feature graph is acquired .Pointwise convolution. The deep-wise convolution only adjusts the dimension of the property graph from the previous layer, while the number of channels does not change, which demands to be modified by pointwise convolution. The pointwise convolution is a 1*1 convolution. Since deep-wise convolution does not merge information between channels, it needs to be applied together with pointwise convolution. It could raise or reduce dimension of feature graphs. The operation of pointwise convolution is similarly to the conventional convolution operation, which is to produce a new attribute map by weighted addition of the property map of the former stage in the depth direction. Each filter exports one feature map, so several channels require multiple filters .3Accuracy is the basis of all applications. Only with better recognition accuracy can convolutional neural networks be widely utilized in rice disease recognition. Small and scattered rice disease sites in the images can lead to poor identification accuracy. To solve this problem, many researchers have started to improve the models in terms of their feature extraction capability and training methods. To enhance the feature extraction capability of the model, multi-scale features of rice diseases are often extracted from the images and merged by introducing an attention mechanism. The improvement of the model training method is mainly implemented by using ensemble learning.By introducing the attention mechanism into a model, it can better understand the correlation between the input data, and improve its prediction ability. Shuai Feng et al. studied the detection of rice blast based on spectral characteristics . The conMultiscale feature fusion can help a model locate the position of the target more accurately. By fusing features with different sensory fields, the model can find the boundary and detailed information of the target, thus improving the accuracy of target detection and localization. Ching-Ling Wang et al. conducted a series or parallel operation of multiple convolutional neural networks to find the best network structure for rice disease recognition . Each inThrough transfer learning, a model can borrow features, relationships, or patterns from the source domain and apply them to the target task, thereby improving the model’s performance and generalization on the target task. Sudhesh K.M et al. applied transfer learning technology and dynamic pattern recognition decomposition method to identify rice diseases . Among m4Rice disease recognition based on convolutional neural networks has obtained very good consequences in the laboratory, and the accuracy is particularly high. However, so far, the application in the process of rice cultivation is still relatively small. One of the most important reasons is that the convolutional neural network employed in the recognition process is still too complex, has too many parameters and high requirements on hardware. Therefore, further lightweight research on convolutional neural networks is needed to make it run on mobile devices such as smartphones.It is very effective to process images through the deep neural network model to complete the classification and regression tasks. However, the parameters of the deep neural network model are generally tens of millions, even hundreds of millions. Such a complex model places high demands on both hardware and software computing resources during training and deployment. This is the reason that there are fewer examples of neural network technology application of rice disease identification. Currently, the lightweight operations of neural network models in rice disease identification process are achieved by adjusting the neural network structure.Igor V. Arinichev et al. tested the performance of four relatively lightweight classical convolutional neural networks for rice disease identification: GoogleNet, ResNet-18, SqueezeNet-1.0, and DenseNet-121 . AlexNetYibin Wang et al. added the attention mechanism and Bayesian optimization algorithm to a convolutional neural network to reduce its volume and finally realized the application of rice disease recognition on mobile devices . The attPoornima Singh Thakur et al. investigated a lightweight convolutional neural network with 7 convolution layers for rice disease recognition named VGG-ICNN . The netRuoling Deng et al. developed a deep learning program based on a convolutional neural network that could run on a smartphone and identify rice diseases . The tra5It is greatly significant to improve the speed of rice disease recognition and realize real-time applications. A lot of researchers have done much work on this aspect and have achieved good results. Rice disease identification based on specific types of images and the use of specific algorithms in the identification process can improve the speed of rice disease identification.Many studies have demonstrated that identification based on rice disease images obtained by hyperspectral techniques and terahertz imaging is often faster. Lei Feng et al. introduced hyperspectral imaging technology and transfer learning into the process of rice disease detection and realized rapid recognition . HyperspOf course, the time required for identification can also be reduced if the common rice disease images are properly processed. Santosh Kumar Upadhyay et al. devised a detection architecture with a convolutional neural network to obtain rapid diagnosis of rice diseases . From thResearchers also found that the recognition model based on the Faster R-CNN architecture was better than other algorithmic models in terms of rice disease recognition speed. An area based convolutional neural network was proposed by Bifta Sama Bari et al. to achieve real-time recognition of rice disease . The enh6Although convolutional neural networks have achieved a lot of results in rice disease recognition, there are still some problems that restrict the development of this research. It is expected that the key work of the research in the future will focus on solving these difficulties.6.1Although rice disease recognition based on convolutional neural networks has achieved good results, it still faces the problems of excessive training data, limited accuracy, and high computer computation consumption in practical applications .Rice disease recognition based on convolutional neural networks and images requires a large amount of data to train the model and validate its performance. Different types of rice diseases may behave similarly on images, and a large number of images must be available to ensure the effectiveness of recognizing different diseases . The con6.2Rice disease recognition based on convolutional neural networks still has vast research space and application value in the future. First, rice disease identification based on small sample data may be a hot spot in the future research. Enhancing the model’s ability of extracting rice disease features by modifying the structure of convolutional neural networks can improve the training speed and extend the application range. Expanding the small sample dataset by data augmentation techniques or employing transfer learning methods to enhance the extraction ability of specific features by convolutional neural networks are the two feasible approaches . Second,7Rice is one of basic foods in the world, and it is greatly significant to ensure its growth and yield security. From growth to harvest, rice is threatened by a variety of diseases, which poses a serious challenge to human food security. To prevent and control rice diseases, human beings began to study and control rice diseases from a very early time. Traditionally, farmers judge rice disease types based on the existing experience, which takes a lot of trouble and effort, and there are serious defects in recognition accuracy and large-scale promotion. As a very successful branch of deep learning technology, the convolutional neural network has made remarkable achievements in image-based classification, recognition, and segmentation. Automatic recognition of rice disease based on convolutional neural networks has also gained many effects in recent years. This paper summarizes the excellent achievements of convolutional neural networks in rice disease recognition in recent years and focuses on its research progress in recognition accuracy improvement, recognition speed optimization, and mobile terminal deployment. The current research on rice disease recognition based on convolutional neural networks mainly focuses on the improvement of accuracy, and the recognition accuracy in the laboratory has reached a high level through data preprocessing and network structure optimization. To extend the application of convolutional neural networks for rice disease recognition in practice, it is necessary to improve its recognition speed and the ability of deploying in mobile devices. Therefore, the simplification of the network structure is also the focus of the research. As convolutional neural networks require a large amount of sample data to support the training process, the needs for computer computation power are also relatively high, which limits the application of convolutional neural networks in rice disease identification. In the future, the convolutional neural network training based on small sample data combined with transfer learning technology, performance optimization of convolutional neural networks combined with ensemble learning technology, and simplification of the network structure for mobile deployment and real-time detection are the focus of convolutional neural networks in rice disease recognition. It is believed that this paper is important for further boosting the capability of convolutional neural networks in rice disease recognition and expanding the utilization of convolutional neural network in rice disease recognition.HN: Writing – original draft, Writing – review & editing. SL: Writing – original draft, Writing – review & editing. QZ: Writing – review & editing. TZ: Conceptualization, Writing – review & editing.


Unimodal biometric systems rely on a single source or unique individual biological trait for measurement and examination. Fingerprint-based biometric systems are the most common, but they are vulnerable to presentation attacks or spoofing when a fake fingerprint is presented to the sensor. To address this issue, we propose an enhanced biometric system based on a multimodal approach using two types of biological traits. We propose to combine fingerprint and Electrocardiogram (ECG) signals to mitigate spoofing attacks. Specifically, we design a multimodal deep learning architecture that accepts fingerprints and ECG as inputs and fuses the feature vectors using stacking and channel-wise approaches. The feature extraction backbone of the architecture is based on data-efficient transformers. The experimental results demonstrate the promising capabilities of the proposed approach in enhancing the robustness of the system to presentation attacks. Biometrics, or human measurements, are metrics related to unique person characteristics that distinguish one individual from another. These biological characteristics can be used as biometric identifiers to recognize and verify a person’s identity. Physiological identifiers include fingerprints, face, palm, iris, and hand geometry, while behavioral identifiers include characteristics such as gait, hand-written signature, and voice. Biometrics technology involves measuring and processing these distinctive biological traits of individuals. Biometric authentication, on the other hand, is a security process that relies on these unique features to verify a person’s identity and ensure that they are who they claim to be.The selection of a specific biometric identifier for a particular application depends on various factors such as uniqueness, permanence, universality, measurability, and performance . FingerpRecently, the Electrocardiogram (ECG) signal of the heart has been proposed as a biometric trait. It has been coined the “heartprint” , and simThese features make it suitable to combine heartprints with traditional biometric modalities like fingerprints, which is the main motivation for the method in this paper. Compared to unimodal biometric systems, multimodal systems are a combination of two or more biometric traits for improved recognition rate and protection against spoof attacks ,20,21,22Recently, a limited number of attempts have been made to use deep learning for the fusion of these two modalities. Hammad et al. ,32 fusedIn this paper, we propose a multimodal biometric solution which combines fingerprint with heartprint modalities. Due to the lack of a multimodal dataset that combines fingerprints and heartprints, we made our own by combining a mini-LivDet2015 dataset and the newly released Heartprint dataset . We usedThe feature vectors extracted by each branch are fused using two approaches: stacking or channel-wise fusion.We propose an end-to-end deep learning model for a multimodal biometric method using image transformers.The proposed method employs a channel-wise fusion approach which improves performance compared to the typical stacking approach.Our contributions via this paper can be summarized in the following points:The remainder of the paper is organized as follows. The purpose of typical fingerprint PAD is to identify whether a fingerprint image is an artefact (PA) or a bona-fide fingerprint sample. Given the fingerprint dataset, As shown in Fingerprints, part of the dermatoglyphics field, are a complex and unique pattern of curving line structures called friction ridges. The number, shape , and location of each ridge make every person unique, and they do not vary with growth or age. The fingerprint image consists of dark lines called ridges and white lines called valleys. In this work, we consider the International Fingerprint Liveness Detection Competition 2015 (LivDet 2015) dataset.The Heartprint2022 database contains a different number of heartbeat samples with a length of 740 each for 199 persons. We propose to extract features from the heartprint and convert it to an image, which is needed for the fusion process. As shown in The third sub-block in the heartprint branch is 2D CNN architecture converts the STFT input image of the size The first fusion strategy is based on the concatenation of the extracted features of the fingerprint image and the heartprint signal. The architecture of the two-branch neural network is illustrated in Multimodal biometric systems seek to increase performance that may not be possible by using a single biometric indicator by providing multiple shreds of evidence of the same identity. An optimal fusion of multiple modalities is a fundamental request for the development of a reliable solution. In attempting to improve the performance of the detection system, the outputs of the fingerprint and heartprint branches are further processed using a fusion module. This fusion module is performed by intercalating the heartprint SIFT image as additional bands to the fingerprint image. We call this a channel-wise fusion approach and illustrate it in A wide variety of deep learning strategies have been used to build biometric identification systems. Usually, these methods depend on CNN to extract features from input data. Inspired by the biological systems of humans, the attention mechanism has revolutionized the natural language processing and computer vision systems ,2. The aThe input image to the ViT is split into Introduced by Facebook AI, the structure of the Deit model, built based on the ViT model , showed To evaluate the proposed method, we first built our own multimodal dataset. Then, we split the dataset into training and testing sets, by splitting based on the subjects.The International Fingerprint Liveness Detection Competition 2015 (LivDet 2015) and the real heartprint dataset, called Heartprint2022, are used to evaluate and validate the performance of the proposed deep learning architecture. The LivDet 2015 dataset is provided by Orrù et al. and can be downloaded from , whereasThe LivDet 2015 dataset has approximately 19,000 fingerprint images captured using four different optical fingerprint sensors: GreenBit, Biometrika, Digital Persona, and CrossMatch . It aimsLivDet 2015 datasets contain spoof fingerprint images collected using artificial fingers. Artificial fingers are fabricated using plasticine-like material to create a negative impression or a mold of the real finger (cooperative method), the mold is then filled to produce the artificial finger using gelatin, PlayDoh, or silicone. A latent fingerprint left on a surface is another way to make artificial fingerprints (non-cooperative method); a transparency sheet is obtained from a processed latent fingerprint and used to create the mold. LivDet 2015 dataset contains spoof images made using diverse materials, such as Ecoflex, gelatin, latex, wood glue, liquid Ecoflex, and RTV (a two-component silicone rubber), as shown in The heartprint dataset is collected using the ReadMyHeart ECG device, by DailyCare BioMedical . The ReaThe non-existence of public multimodal datasets encompassing heartprint and fingerprint signals incited us to build a multimodal dataset from the LivDet 2015 dataset and Heartprint dataset. To this end, we first detect the heartbeats from each heartprint record in Heartprint using an effective curvature-based technique . Then, tWe used a subset of 70 subjects from the LivDet 2015 dataset, each of which has 10 bona-fide images from the Digital Persona sensor and 12 artefact images to construct a mini-LivDet 2015 dataset. Then, we associated a random subject from the Heartprint2022 dataset to each subject from the mini-LivDet 2015 dataset. If the fingerprint image is fake, the subject must have different heartprints. To substantiate this fact, we assign a random set of heartbeats from the same subject to the bona-fide images of one person, and we assign the random set of another randomly selected person to the artefact fingerprint of the same person. The new multimodal dataset containing fingerprint images and heartbeat signals of 70 subjects is described in During the training phase, the model receives data batches consisting of both classes, the bona-fide fingerprint samples with bona-fide heartbeat samples and fake fingerprint samples with bona-fide heartbeat samples of another person. In this way, the deep model learns to discriminate between the fake fingerprint sample and the bona-fide fingerprint sample.We conducted different experiments to evaluate the performance of the proposed methodology and compare our results with previous state-of-the-art methods. In the first experiment, we trained the deep model using only the fingerprint branch to assess its performance regarding the detection of PAs. The model is trained using the training subset of the LivDet 2015 dataset. Then, we trained and evaluated the model on the multimodal dataset, using the proposed fusion strategies to detect and prevent PAs.To check the reproducibility of the proposed deep learning model, all experiments are repeated several times and the average classification accuracy is reported. All experiments are implemented in Python and the PyTorch machine learning library using a workstation that has a Core i9 processor with a speed of 3.6 GHz, 64 GB of memory, and GPU with 11 GB GDDR5X memory. In the first experiment, we trained different deep models on the multimodal dataset. The average detection accuracy of the different models using a single modality (no fusion) and multimodality (with fusion) biometric traits is reported in We note from the reported results in In the second experiment, we applied the fusion between the fingerprint and the heartprint signal at the data level. We combined the two modalities by inserting the heartprint image as a new channel in the fingerprint image. We trained different pre-trained models on the combined fingerprint-heartprint to extract significant features from the combined images.As shown in A.Sensitivity Analysis of the Number of Training SubjectsGenerally, it is common knowledge that a small training dataset produces weak approximation . To asseThe reported results in B.Sensitivity of the heartprint featureDuring this experiment, we assess the effect of the number of heartbeats used in the STFT heartprint image on the model performance accuracy. We repeated the experiment with STFT heartprint images built using a different number of heartbeats (heartprint of length ranged between 5 heartbeats and 20 heartbeats), the model performance is reported in In this research, we have presented an end-to-end deep learning multimodal fusion technique for a fingerprint presentation attack detection system. We introduced a variety of pre-trained deep learning models as feature learning backbone networks. The selection of an appropriate modality and the choice of an optimal fusion strategy are crucial factors in the performance of a biometric system. To enhance the performance of the PAD system, the heartprint is added as an additional modality, and two fusion strategies, i.e., heartprint-fingerprint feature concatenation and heartprint-fingerprint channel insertion, are adopted and tested. The obtained experimental results show that the fingerprint-heartprint multimodal PAD system achieved high accuracy, confirming the effectiveness of the proposed approach. For future developments, we propose to investigate other fusion strategies based on feature selection, feature weighting, and decision fusion to increase the robustness of the system.


Urban mobility is facing many challenges, such as energy consumption, pollution, and safety. Therefore, it is necessary to analyze the mobility of users through the transportation network using data containing information regarding the used transport mode. This data article describes a dataset from mobile devices collected by users as they move through the transportation network. Each sample in this dataset is labelled with a corresponding transport mode. Eight transport modes are present in the dataset: Car, Bus, Walking, Bicycle, Train, Tram, Running and Electric Scooter. The basic breakdown of the raw data according to users, transport modes and multimodal routes is presented. During data collection, data from the accelerometer, magnetometer, and gyroscope sensors mounted within the mobile device were stored. The data were collected using a mobile application from mobile devices with an embedded Android operating system. The structure of the text files in which the data were stored and the structure of the application used to collect the data are presented in the paper. The collected data provides a highly relevant basis for mobility analysis and planning, analysis of road conditions, clustering of user behaviour, and comparison of transport mode classification methods. Specifications Table1•The dataset increases the diversity of data in the area of multimodal transport activities and thus enables the improvement of existing methods for the classification of transport modes.•Researchers in the field of Human Activity Recognition (HAR) and transport mode classification can compare existing and newly developed methods.•The dataset contains raw sensor data for the e-scooter transport mode, which is not included in published datasets.•The dataset can be used to extend the benchmark problem to additionally test spatial dependencies of transport mode classifiers.•The dataset can be reused for analysis of sensor-based multimodal travel activities, trajectory segmentation, mobility analysis and planning, analysis of road conditions or clustering of user behaviour.2The current dataset is created to establish a benchmark for comparing transport mode classification methods. Additionally, it aims to enable spatial independence comparisons between different methods by utilizing more datasets collected from distinct transport networks. A dataset marked with transport modes is important in conducting a thorough mobility analysis, which can provide insights into different transport modes, enabling transportation network improvement with a focus on sustainable transport modes.3Unknown. Samples are assigned the label Unknown when the transition between two transport modes takes too long. Out of 83227415 records in the dataset 37 of them had the label of the transport mode Unknown. These records are considered unreliable and are typically excluded from most research analyses. Additionally, records that lacked any recorded sensor measurements were entirely removed and are not included in the published dataset. It is important to note that all remaining data are presented in their raw form, without applying any filtering methods to address potential sensor errors in the measurements to accurately represent real-time conditions and ensure the authenticity of transport mode detection. This approach is crucial when developing a real-time model for transport mode detection, as it allows simulation of data arrival and simultaneous identification of the transport mode.The raw sensor data is organized within text files, with each file being designated a unique name in the format of UserID_TrajectoryID. The dataset comprises 454 text files, representing 454 distinct trajectories. Each trajectory is marked with a flag indicating user confirmation of the route. Only routes that have been confirmed by the user are included in the published dataset. Each text file comprises 18 columns of data, as detailed in The dataset was collected during a 5-month period, spanning from November 2021 to March 2022. The quantity of data collected for various transport modes is not uniform, as illustrated in A more detailed distribution of data can be found in Haversine formula was employed to compute the geographical distance between two points in the trajectory, which calculates the shortest distance between two points on the surface of the Earth sphere Consequently, users who primarily use non-motorized transport modes cover shorter distances than those who primarily use motorized modes. For instance, when comparing users with IDs 18 and 37 in terms of time traveled, user 18 has more hours of collected data. However, the data collected from user 37, when expressed in terms of distance traveled, is approximately 60% greater than that of user 18.The duration of each trajectory varies and there are distinctions in trajectory duration among users, as illustrated in The trajectories can also be observed considering the number of transport modes used. 41.activate the mobile application while navigating through the transportation network.2.select the transport mode being used.3.verify the accuracy of the acquired geographic data on the digital map as per the provided instructions.4.deactivate the mobile application upon completion.Data collection was conducted using a mobile application developed for devices with the Android operating system. Participants in the data collection were provided instructions on how to use the application upon installation on their personal mobile devices. During data collection, participants were prompted to:The proposed mobile application has been designed to function in a background mode, enabling concurrent use of other applications or screen lock on the device. A notification mechanism has been integrated to inform the user of the background activity. Users are given three key pieces of information about the transport mode classification: the time of the transport mode change, the standing classification, and the route summary, before being prompted for confirmation. Additionally, users are required to give their consent for data collection. To ensure users mark transport mode changes consistently, instructions are provided for users to make the transport mode change at the time of initiating use of the new transport mode. The second instruction pertains to standing, where users are instructed that standing is considered a part of walking . To validate the correctness of the trajectory, users should examine several segments of the trajectory. The first check involves the accuracy of the geographic positions of the displayed trajectory. The second check involves the verification of the transport modes used, i.e. whether the transport mode symbols on the digital map match the transport mode used. The final check pertains to the geographic locations of transport mode changes, where the user must confirm that the transport mode was actually changed at these locations.Collecty” comprises of 6 consecutive activities, as shown in The mobile application “Change mode button) or end the route (by pressing the End route button). When the user presses the button to change the transport mode, the application returns to the previous activity where the user can select the transport mode again. Another option is to complete the trajectory, which leads to the last activity. In the last activity, the user can verify the route on the digital map. The OpenStreetMap (OSM) was used to display the route, Confirm route button) or reject the route (by pressing the Reject route button). Based on the user’s decision, all records of the displayed route will be classified as 1 (confirmed route), 2 (rejected route) or 3 (no decision made). GPS points are plotted on the digital map every 5 s or if more than 100 m have passed since the last GPS point. After user confirmation, a message is displayed and a button to exit the application (Exit app), as shown in The third activity refers to monitoring the parameters from the sensors of the user’s mobile device and is used solely to visualize the data for the user. In the background, data are sent to the server or stored in the internal memory of the mobile device. As can be seen in the Relevant informed consent was obtained from participants involved in data collection process and the participant data has been fully anonymized.Martina Erdelić: Conceptualization, Data curation, Software, Validation, Formal analysis, Writing – original draft, Writing – review & editing. Tomislav Erdelić: Software, Data curation, Formal analysis, Writing – review & editing. Tonči Carić: Supervision, Writing – review & editing.


Bio-inspired flying robots (BIFRs) which fly by flapping their wings experience continuously oscillating aerodynamic forces. These oscillations in the driving force cause vibrations in the motion of the body around the mean trajectory. In other words, a hovering BIFR does not remain fixed in space; instead, it undergoes oscillatory motion in almost all directions around the stationary point. These oscillations affect the aerodynamic performance of the flier. Assessing the effect of these oscillations, particularly on thrust generation in two-winged and four-winged BIFRs, is the main objective of this work. To achieve such a goal, two experimental setups were considered to measure the average thrust for the two BIFRs. The average thrust is measured over the flapping cycle of the BIFRs. In the first experimental setup, the BIFR is installed at the end of a pendulum rod, in place of the pendulum mass. While flapping, the model creates a thrust force that raises the model along the circular trajectory of the pendulum mass to a certain angular position, which is an equilibrium point and is also stable. Measuring the weight of the BIFR and the equilibrium angle it obtains, it is straightforward to estimate the average thrust, by moment balance about the pendulum hinge. This pendulum setup allows the BIFR model to freely oscillate back and forth along the circular trajectory about the equilibrium position. As such, the estimated average thrust includes the effects of these self-induced vibrations. In contrast, we use another setup with a load cell to measure thrust where the model is completely fixed. The thrust measurement revealed that the load cell or the fixed test leads to a higher thrust than the pendulum or the oscillatory test for the two-winged model, showing the opposite behavior for the four-winged model. That is, self-induced vibrations have different effects on the two BIFR models. We felt that this observation is worth further investigation. It is important to mention that aerodynamic mechanisms for thrust generation in the two and four-winged models are different. A two-winged BIFR generates thrust through traditional flapping mechanisms whereas a four-winged model enjoys a clapping effect, which results from wing-wing interaction. In the present work, we use a motion capture system, aerodynamic modeling, and flow visualization to study the underlying physics of the observed different behaviors of the two flapping models. The study revealed that the interaction of the vortices with the flapping wing robots may play a role in the observed aerodynamic behavior of the two BIFRs. Having understood the lifting mechanisms in insect flight, several researchers independently developed unique designs for FWMAVs5. Zakaria et al. (2015) showed that the inclusion of inertial power requirements is essential for physical and proper optimization6. They further studied the aerodynamic forces generated in forward flight for different Reynolds numbers and flapping frequencies7. Whitney8 designed and developed one of the tiniest FWMAVs. Keennon et al.9 developed the AeroVironment Nano Hummingbird as a hovering ornithopter. TU Delft researchers designed, developed, and studied the aerodynamic and dynamic performances of the Delfly11. Any design, like the previously mentioned ones, is aimed at a particular objective or flying condition like forward flight, hovering, etc.Bio-inspired flying robots (BIFRs), more specifically Flapping Wing Micro Air Vehicles (FWMAV), have been a major focal point of research in the aerodynamics, dynamics, and control community in the last few decades. In the twentieth century, the main attention was directed toward uncovering the unconventional lift mechanisms in flapping flight. With the more precise observation of the insect flight and how they make use of the unsteady lifting mechanisms , this puzzle was resolved12. However, Ellington13 examined the results of Weis-Fogh’s in the light of more accurate kinematic and morphological data, and his conclusion was opposite to that of Weis-Fogh’s. Ellington14 further showed that the leading edge separation bubble plays a prominent role in the hovering flight of insects. In comparison to a thin airfoil, Ellington asserted that the leading edge bubble modifies the camber and the thickness of the thin profile which enhances lift at low Reynolds numbers. Bayiz et al.15 compared the hovering efficiency in rotary and flapping modes using rigid rectangular wings. They observed that flapping wings are more efficient in achieving a higher average lift coefficient in hovering. Sarkar et al.16 studied aerodynamic performance under asymmetric flapping kinematics using sinusoidal and triangular waveforms. The frequency-asymmetry mechanism showed an increase in aerodynamic loads for the sinusoidal case. During the faster stroke, the lift can be enhanced depending on the level of asymmetry. The results of these investigations can be used to design an efficient flapping robot for hovering.One of the most challenging flying conditions of flapping flight is hovering. There have been numerous studies to investigate the aerodynamics of hovering flapping flight. Weis-Fogh tested the quasi-steady assumption for insect flight, where unsteady effects are more conspicuous, and showed that quasi-steady aerodynamics can predict the main features of hovering flights17 found that the pitching moment produced by a change in horizontal speed is the primary source of an unstable oscillatory mode, whereas vertical force produced by changes in vertical speed is the primary source of a stable slow subsidence mode. These results are mainly based on averaging the flight dynamics over the flapping cycle. In contrast, using chronological calculus a hidden stabilization mechanism was discovered in a hovering hawkmoth. Taha et al.18 showed that insects use a passive stabilization mechanism through their natural wing oscillation; this is called vibrational stabilization. It is a natural phenomenon observed in systems like the Kapitza pendulum19. A bio-inspired flapping robot in two degrees of freedom system can exhibit vibrational stabilization as well20.Discussion on hovering insects is incomplete without pondering upon the question of stability. Sun et al.21. Hence, it experiences self-induced vibrations. This vibration may change the flow field around the wing and thus affect the generated aerodynamic forces. In ideal hovering, there should be no self-induced vibration but in real cases, these vibrations are unavoidable due to the inevitable oscillatory nature of the driving aerodynamic forces. To the authors’ best knowledge, there is little effort exerted that focuses on this point23, and the effect of self-induced vibration on the clapping mechanism is significantly under-explored. The current work is dedicated to studying the effect of self-induced body vibration in the flapping flight. To achieve this goal, we considered two different setups for aerodynamic force measurement  : (1) Pendulum setup or oscillatory test and (2) Load cell setup or fixed test. In the pendulum setup, we replaced the mass of a pendulum with a flapping wing robot. Whenever the robot flaps, it generates thrust and moves upward along the circular trajectory of the pendulum, assuming equilibrium at some angular position. We can measure this angle and use it to calculate the average thrust. We prefer this pendulum setup over the Harvard Robofly experimental setup 24 because it allows multiple equilibrium positions at different flapping frequencies. Hence, the thrust can be estimated via a simple measurement of the angular position of the pendulum rod. In contrast, if the thrust of a flapping robot moving along vertical rails is different from its weight, the robot will continuously move up (or down); and estimating the thrust will require accurate measurement of the acceleration/deceleration of the robot. Clearly, measuring the angular position of the pendulum rod is simpler and more accurate than measuring the acceleration of a vibrating body. Also, this is a standard setup used to measure thrust generated by rotary wings25. The flapping robot vibrates about the angular position i.e., the measured thrust includes the effect of vibration. In contrast, in the loadcell or fixed test setup, there is no room for such a vibration. So using these two setups, we can measure the effect of self-induced vibration on flapping thrust generation.It is important to emphasize that when an insect hovers over a flower, it is not completely stationary in space over the flower. Instead, it undergoes oscillatory motion in almost all directions. These oscillations can be observed in the video, Hedrick and Daniel (2006) presented in the supplementary section of their paper1, wake capture26 are utilized by the two-winged model in flapping flight. The four-winged model or Model B  to generate thrust. Outcomes of wing-wing interaction like stronger leading edge vortex33 and jet effect34 are utilized for thrust generation by the clap-and-peel mechanism. Balta et al.35 showed with flow visualization that the peeling phase of the flapping cycle draws air in and the clap phase propels it downstream; that thrust is augmented using a jet effect. A blob of air flows between the wings in the peeling phase, which strengthens the leading edge vortex. Some efforts were made to capture the clapping effect into an aerodynamic model38. Armanini et al.33 studied the improved strength of the leading edge vortex during clapping and included this phenomenon in a quasi-steady aerodynamic model. They further studied the clapping-effect interaction with the tail-wake and wing-wake32.Two flapping wing robots are considered for this study: one has two wings and the other has four wings. The one, that is two-winged, is named Model A Figs. a, 3a, 4.mentclass2pt{minim34. Thus it is expected that the effects of the self-induced vibration will not be similar for the two-winged and the four-winged robots.The mechanism for thrust generation in a four-winged robot is different from the conventional one due to the wing-wing interaction, or the clap-and-peel mechanismIn the present work, we investigated the effect of self-induced body vibration on two-winged and the four-winged (clap-and-peel) flapping robots using two different thrust measurement setups, flow visualization, motion capture, and theoretical aerodynamic modeling.35. In the case of Model B, wing-wing interaction takes place which exploits the clap-and-peel mechanism. The motion of the wings moving towards each other during the flapping cycle is called ’clap’, and the ensuing motion of moving away from each other is known as ’peel’. Given the flexibility of the wings, they peel away in this motion and hence the name. Figure The focus of this research is to investigate the effect of self-induced oscillations on two distinct mechanisms of thrust generation for FWMAVs. The flapping robots, Model A Figs. a, 2a, 3a39.The present work is dedicated to testing and analyzing Models A and B. We use different experimental setups like (1) Pendulum test or oscillatory test Fig. a and (2)m) we can calculate the average thrust over the flapping cycle. We obtained equation , i.e., no wing-wing interactions. Armanini et al.33 extended the applicability of this model to FWMAVs with four wings (Model B), by including the clapping effect.To further understand the effect of perturbation velocity on flapping wing models (Model A and Model B), it may be prudent to develop an aerodynamic model for both models, which is the focus of this section. The backbone of the adopted aerodynamic model was proposed by Berman and Wangr from the body of the FWMAV. The red portion represents the chord of the section c(r) and  makes the reference frame at the leading edge (denoted as LE) of the blade element. The green and blue arrows at the leading edge denote the direction of perturbation velocity  x and y directions, 41 the forces r (spanwise location) and t (time). The pitching angle is modeled as a Fourier series in time since the motion is periodic, with spatially varying coefficients to account for wing flexibility.Figure equation .7\documeAfter obtaining thrust per unit span equation , it is tentclass1pt{minimaUpon defining the complete structure of the aerodynamic model, some unknown parameters must be specified. These include mentclasspt{minima33.In the case of the four-winged robot, the aerodynamic model is almost the same as that of the two-winged robot with a few extensions to capture the effect of wing-wing interaction. During the ’peel’ motion, a suction is created between the wings, which sucks air from the ambient towards it and strengthens the leading edge vortex. Therefore, the generated circulation is modified in a way that empirically captures the change in the strength of the leading edge vortex.Hz) to present a more complete picture than just inspecting thrust measurements. We scrutinize the flow field using flow visualization along with measuring the oscillatory motion of the body and feeding it into the presented aerodynamic model to gain some insight into the observed behaviors presented in Fig. Hz case will hold exactly at other frequencies. However, we expect the pictures to be reasonably similar. The spanwise sections of 25% and 35% of the wing are chosen for flow visualization of Model A. The sectional flapping velocities at the above mentioned locations are presented simultaneously with the perturbation velocity at a given point during the flapping cycle. For Model B, 15% spanwise position is chosen and similar flow field images and velocities are investigated. Reference speed In this section, we focus on the behavior of the flapping models at one flapping frequency (6Hz flapping frequency (f). The row presents flow fields at a given time during the flapping period. The time parameter is denoted by t, whereas The images presented in Fig. es Figs.  and 19 rFigure 41 without any perturbation, while 42). Due to this response, the closer the TEV is to the trailing edge, the smaller its strength is compared to the steady value. Figure Figure 35, the four-wings mechanism generates thrust through the clap and peel mechanism. During the peel motion, it creates a suction between the wings, which intakes a significant amount of air. During the clap motion, it pushes the air downstream creating a ’jet burst’. Figure This sub-section is dedicated to analyzing the effect of self-induced vibration in the flow field of Model B at a 15% spanwise location. As mentioned by Balta et al.In table 23. This interaction is not captured in the pendulum setup because the stroke plane is almost parallel to the pendulum rod; the backward body motion  is restrained. From this discussion, it may be concluded that the measured thrust forces in the two-winged model, pendulum-setup case are higher than those in free flight, which does not contradict the general conclusion of the study: body oscillations reduce the averaged thrust in the two-winged model. In the four-winged model, however, the pendulum constraint may not be as restrictive because the instantaneous drag and inertial forces coming from each wing may cancel when transferred to the body (as they move in opposite directions).Finally, it may be prudent to emphasize that the obtained results and conclusions might not be directly extended to biological flyers or FWMAVs in free flight. The constraints in the pendulum setup may limit body motion, which, if allowed, could decrease the resulting aerodynamic loads (particularly for the two-winged model): A flapping wing experiences drag as it sweeps back and forth; this drag is transferred to the body, which causes the body to move opposite to the flapping wing. As such, the speed of the flapping wing with respect to the surrounding quiescent air is less (it is the flapping speed minus the body’s backward speed), which results in smaller aerodynamic loadsThe current study compares two cases of hovering flights. The first case is of an ideal hovering; there is no room for perturbation. The measurement in this case is carried out using the loadcell setup, which is also called the fixed test. The other case of hovering allows for concomitant self-induced vibration due to the oscillatory nature of the thrust force. A pendulum setup, which is also called the oscillatory test, is used to measure the average force in this case. This force measurement includes the effect of the self-induced vibration. These cases are studied using two different flapping wing robots: a two-winged robot or Model A and a four-winged robot or Model B. When these models are tested in the above-mentioned setups, it is observed that the fixed test measures more thrust than the oscillatory test for Model A. The opposite behavior is observed for Model B as shown in Fig. 41) for thrust generation are used to match the loadcell data by optimizing some unknown parameters. The perturbation velocity, measured using the motion capture system, is applied in the model to study the effect of the induced vibration during the flapping cycle. The perturbation is believed to have some effect on the flow field. To investigate how much impact the vibration has on the flow field, the flow visualization technique is used to look into it. The aerodynamic modeling and the flow visualization are done at the 6Hz flapping frequency.Two well-known aerodynamic models (Flow visualization revealed some interesting vortex interactions. In the case with no vibration, Model A enjoys a certain jet effect near its trailing edge. The perturbation wanes its effect by moving the whole flapping robot into the jet. This decreases the overall thrust for Model A in the oscillatory test. For Model B the self-induced vibration enhances the thrust by moving the flapping robot away from the jet. This phenomenon enhances the clapping effect and consequently increases the overall thrust in the oscillatory test. The vortex interactions show how the self-induced vibration has an adverse effect on the thrust generation for a two-winged flapping robot. In contrast, the vibration enhances the effect on a four-winged flapping robot.Supplementary Information.