Information-theoretically provable unique true random numbers, which cannot be correlated or controlled by an attacker, can be generated based on quantum measurement of vacuum state and universal-hashing randomness extraction. Quantum entropy in the measurements decides the quality and security of the random number generator (RNG). At the same time, it directly determines the extraction ratio of true randomness from the raw data, in other words, it obviously affects quantum random bits generating rate. In this work, we commit to enhancing quantum entropy content in the vacuum noise based quantum RNG. We have taken into account main factors in this proposal to establish the theoretical model of quantum entropy content, including the effects of classical noise, the optimum dynamical analog-digital convertor (ADC) range, the local gain and the electronic gain of the homodyne system. We demonstrate that by amplifying the vacuum quantum noise, abundant quantum entropy is extractable in the step of post-processing even classical noise excursion, which may be deliberately induced by an eavesdropper, is large. Based on the discussion and the fact that the bandwidth of quantum vacuum noise is infinite, we propose large dynamical range and moderate TIA gain to pursue higher local oscillator (LO) amplification of vacuum quadrature and broader detection bandwidth in homodyne system. High true randomness extraction ratio together with high sampling rate is attainable. Experimentally, an extraction ratio of true randomness of 85.3% is achieved by finite enhancement of the laser power of the LO when classical noise excursions of the raw data is obvious. Randomness is one vital ingredient in modern information science, in the regime of both classical and quantum ,2, sinceEntropy is defined relative to one’s knowledge of an experiment’s output prior to observation. The larger the amount of the entropy, the greater the uncertainty in predicting the value of an observation. Among types of entropy, min-entropy is a very conservative measure. In cryptography, the unpredictability of secret values is essential. The min-entropy measure the probability that a secret is guessed correctly in the first trial. For mathematically determining min-entropy of a secret, the first thing is to precisely identify the distribution that the secret was generated from .Quadrature fluctuation of optical quantum vacuum state, the nature initial state of optical field at room temperature, is the noise source for the random bits generation in this scheme. According to Born’ s rule, the measurement outcome of a pure quantum state can be intrinsically random. A single measurement of the quadrature of the vacuum state is completely random and multiple repeated measurements satisfy the Gaussian distribution statistically, so we can extract random bits from the measurement results. Based on homodyne measurement, the microscopic fluctuations of quadrature of the vacuum state are detected, amplified and transferred into an electric signale system . WithoutThe coefficient hy (OHT) . In thisWhen classical noise is taken into account, such as electronic noise and local noise resulted from imperfect balancing in balanced homodyne detection (BHD), the observed probability distribution of the electric signal is in the form of a convolution of the scaled vacuum state marginal distribution and the classical noise histogramThe vacuum noise and the classical noise as two variables with normal distribution, are independent with each other, thus their sum is also normally distributed with a total variance equal to the sum of the two variances.QCNR) in the homodyne measurement system is defined asAccording to Equations (2)–(4), the homodyne measurement of the vacuum state yields a signal distribution as followsQCNR related to the signal-to-noise ratio of homodyne detection, is defined as the ratio between the mean square noise of the measured vacuum state and the electronic noise, that is, the quantityQCNR of the raw data can be indicated from the clearance shown by spectrum analyzer.The n-bit ADC with a dynamical range n. 3-bit ADC binning is shown in In our proposal, as a continuous-variable, the measurement output consisting of scaled quadrature of the vacuum state and the classical noise is discretized by an In order to design an entropy source that provides an adequate amount of entropy per output bit string, the developer must be able to accurately estimate the amount of entropy that can be provided by sampling its noise source. The behavior of the other components included in the entropy must also be known clearly since the behavior of the other components may affect the assessment of the entropy. In our system, the randomness or the entropy in the measurements could derive from multiple factors, such as the quantum fluctuation, classical influences on it and even malicious attack from the third part . EspeciaIn this scheme, the min-entropy of the probability distribution of quadrature measurements can be accurately predicted from the probability density function of the quantum signal. The maximum probability in (9) can be acquired based on the probability distribution discretized by the binsEach bin is labelled by an integer n-bit sampling properly whenever possible.Secondly, some restrictions must be taken into account in analog-digital conversion process. Those samples go off-scale, that is, points in saturation will be recorded as extrema values as depicted in Further, considering the influence of classical noise on the measurement outcome, ADC dynamical range should be optimized over the classical noise shifted quantum signal probability distribution. In application scenario, inevitable classical noise excursion in the measurement system will result in nonzero mean in the measured signal probability distribution. On the other hand, eavesdropper may induce a deliberate offset over the sampling period. In a word, a noticeable classical noise excursion, Taking into account all these factors offered above, we rewrite the discretized probability distribution as,In the best-case scenario of ADC sampling range, the measurement outcome probability in the center bin is equal to the higher one of the first and the last bins. In this way, the quantum conditional min-entropy is information theoretically provably estimated and the amount of quantum-based randomness in the total noise signal is rigorously evaluates. In applications with the requirement of information security, a random sequence is demanded to be truly unpredictable and have maximum entropy .QCNR and derive the probability distribution of the quantum signal. The higher the QCNR, the more true randomness can be extracted from the raw measurement. Only when QCNR is high enough, both the quality and the security of the random number generator are guaranteed. Fulfilling the condition of optimal dynamical sampling range R, minimum-entropy of the quantum signal for growing clearance is theoretically analyzed. Proceeding from the directly measurable quantity, homodyne clearance, corresponding QCNR is derived from Equation (8). Then quantum noise variances are expressed as multiples of the R is decided based on Equation (11). Finally, based on Equation (13), the quantum conditional min-entropy in optimal sampling range scenario as a function of different classical noise excursion is analyzed.At the same time, the conditional min-entropy sets the lower bound of extractable randomness from the raw measurements and quantifies the least amount of randomness possessed by each sample or QCNR while the classical excursion is subtle. Whereas if classical noise excursion is evidence, one can achieve high secure randomness only when clearance is high enough.The classical noise excursions in our raw data have been collected from multiple measurements, which range from almost 3 to 29 times of classical noise standard deviation QCNR, but also large detection bandwidths, are pursued, since the cut-off frequency of the homodyne detector upper bound the sampling frequency in random numbers generation process [The clearance relies on the total gain in homodyne detection system . As a trade-off, lower electrical gain put up with higher op-amp bandwidth. In fact, theoretically, vacuum quadrature fluctuates with unlimited bandwidth in the frequency domain. The random number generation rate in this scheme is ultimately limited by the bandwidth of the homodyne detector. Increased bandwidth of op-amp allows higher sampling rate.Experimentally, we dedicate to enhance quantum entropy in quantum RNG by enhancing the laser power of LO beam to noise-independently amplify quadrature fluctuation of vacuum state on the premise of optimizing ADC sampling range. An extraction ratio of true randomness of 85.3% is achieved by finite enhancement of the LO power when classical noise excursions of the raw data is obvious and the extracted random sequences passed the NIST , Diehard and the TestU01 tests.The experimental setup is depicted in Classical noise in the photocurrents is rejected effectively over the whole detection band while the clearance has dependence on frequency as shown in In OHT, BHD system is established and locked to every relative phase to measure the marginal distributions of electromagnetic field quadrature for completely reconstruction of quantum states . While tQCNR as a function of the LO power arriving at the PD. The electrical noise variance is relatively consistent for certain TIA gain. The clearance depends only on the LO power. The noise power is given byQCNR. QCNR as a function of the LO power figured out from the measured clearance levels is plotted with open circles in QCNR on LO power. It is shown as the black dashed line in QCNR, which is due to uncertainties in determining the transimpedance of the detector and the transmission losses in the LPF.We present the mpedance . For eacQCNR of 17.8 dB in our system, limited by the maximal output power of the laser. The signal is sampled with a rate of 100 MHz, upper limit of twice the LPF band for the sampling rate to avoid temporal correlation between samples. The resolution is 12 bits and the dynamical range is optimized according to the histogram of the time series acquired with reasonably larger sampling range. The amplitude acquisition scale of oscilloscope  is continuously adjustable. By choosing the analog-digital conversion range appropriately and tuning the LO intensity finely, the amount of off-scale points can be controlled within allowed statistical deviation. The number of saturated points is easy to restrain on-line from the oscilloscope. The distributions of the random data in time domain and in histogram are shown as insets of We increase the LO power up to 6 mW to achieve the largest m final random bits are extracted by multiplying the matrix and n raw bits, where Finally, information-theoretically provable post-processing scheme, Toeplitz-hashing extractor, is constructed on an FPGA to extract true randomness from the raw data and uniform the Gaussian biased binary stream . A binarp-value to measure the uniformity of the multiple p-values. The test is considered successful if all the final p-values lies in the range from 0.01 to 0.99 [Results of the Diehard statistical test suite for the same data file is shown in  to 0.99 .p-value from a failing test converges to 0 or 1. Where the test has multiple p-values, the worst case is tabled in Constrained by the computational power of crush of TestU01, small crush test is performed with a data size of 8 G bits . The ranOn the other hand, we reduce the LO power in the homodyne system to 400 μW and correspondingly, the clearance declines to 4.06 dB. The time series of the system outcomes are collected and statistically analyzed. Classical noise excursion in the Gaussian distribution is about 19.3 times of the classical noise standard deviation. Based on theoretical calculation, the min-entropy is worked out to be 7.73 bits/sample. The hash extraction results with maximum extraction ratio of 0.63 can pass the NIST, Diehard and TestU01 tests finally. QCNR is high enough. With the LO power enhanced, the vacuum quadrature fluctuations are amplified independent of the electrical noise and the quantum entropy content in the raw data is enhanced effectively. Thus, we propose large dynamical range and moderate TIA gain to pursue higher LO amplification of vacuum quadrature and larger detection bandwidth in homodyne system for higher sampling rate in random numbers generation. Higher hash extraction ratio along with higher sampling rate will enhance the real random number generation rate effectively. More importantly, the quantum RNG system is more robust against to the third part attack.To summary, in this work, we discussed the role of LO power plays in random number generation based on quantum detection of vacuum state. When classical noise excursion in the system is trivial, LO power in the homodyne system affect the quantum entropy in raw data insignificantly. Nevertheless, in realistic scenario, the mean of the measured signal distribution is normally nonzero, even much larger noise excursion may be induced deliberately by the eavesdropper. In this case, enough real randomness is attainable only when The central mathematical concept in true RNG is entropy, which is the assessment standard of the security and quality of a RNG. There are many types of entropy. In recent years, min-entropy, a very conservative evaluation, is applied to lower bound the entropy content in quantum RNG and as the indicator for extraction ratio of universal hash extractor. In our work and some ever works ,19, quan12, we take the lower 8 bits to meet the applicability of the test. The resulting min-entropy is taken from the minimum of all the estimates as 5.818 per 8 bits. The restart tests are passed. Although the ratio of 72.7% is lower than the evaluation of quantum conditional min-entropy, the quality of our entropy source is validated.As discussed above, the raw noise-source output in our proposal is biased, Toeplitz hash extractor (conditioning component) is used in the design to reduce that bias to an appropriately level before the RNG exports any bits. For non-IID data, a list of estimators is proposed and the minimum of all the estimates is taken as the entropy assessment of the entropy source for the entropy source validation for the Recommendation. We apply our raw bit strings to the test suit on line . The Tes


T cells are critical for co-ordinating the immune response. T cells are activated when their surface T cell receptors (TCRs) engage cognate antigens in the form of peptide-major histocompatibility complexes (pMHC) presented on the surface of antigen presenting cells (APCs). Large changes in the contact interface between T cells and APCs occur over the course of tens of minutes from the initial contact to the formation of a large-scale junction between the two cells. The mature junction between a T cell and APC is known as the immunological synapse, and this specialized plasma membrane structure is the major platform for TCR signaling. It has long been known that the complex organization of signaling molecules at the synapse is critical for appropriate activation of T cells, but within the last decade advances in microscopy have opened up investigation into the dynamics of T cell surface topology in the immune synapse. From mechanisms mediating the initial contact between T cells and APCs to roles in the organization of molecules in the mature synapse, these studies have made it increasingly clear that local membrane topology has a large impact on signaling processes. This review focuses on the functional consequences of the T cells' highly dynamic and heterogeneous membrane, in particular, how membrane topology leads to the reorganization of membrane proteins on the T cell surface. T cells are integral members of the immune system, in which they mediate adaptive immunity through direct, antigen-specific contact with antigen-presenting cells (APCs). In order to initiate an appropriate immune response, the T cell relies upon external signals communicated to them by proteins on the surface of an APC. The signal given to the T cell determines the effector response, for instance, an APC can trigger the activation of a T cell by presenting peptide antigens on major histocompatibility complexes (pMHC). pMHCs can bind to the T cell receptor (TCR) on the T cell membrane, triggering a signaling cascade through the associated cluster of differentiation 3 (CD3) subunits, which includes recruitment of cytosolic signaling proteins such as the protein kinase zeta-chain-associated protein kinase 70 (Zap70) and phosphorylation of membrane proteins such as the linker for activation of T cells (LAT) , and amphipathic helix insertion, or steric interactions to facilitate energetically favorable binding to the membrane  gave a quantified description of a force generation sequence upon local bidimensional engagement of the TCR-CD3 complex  and collectively form an immunological synapse with an APC  . AlthougThus, the use of various light microscopy techniques have indubitably been an asset to understanding how these complex immune cells function, however, these methods are inherently limiting. Therefore, one should consider how these findings may apply to broader three-dimensional interactions between a T cell and an APC.In physiological settings, T cells make contact with deformable surfaces and must penetrate the thick glycocalyx present on cells. Therefore, protein distribution may vary when the T cell comes into contact with an APC (and may differ from that in a reconstituted system). For these reasons, focus has shifted recently to taking membrane topological features into consideration when activating and imaging immune cells.Various imaging approaches facilitated the finding that membrane protein distribution may be linked to cell topography. One such approach is combining variable-angle TIRF (VA-TIRF) with stochastic localization nanoscopy (SLN) to determine the relationship between TCR membrane distribution and T cell microvilli  were shown to be used by T cells for probing cells for antigens, much like microvilli, penetrating the surface of planar endothelial cells and facilitating trans-endothelial migration in the absence of antigen. This finding is consistent with the theory that TCRs are accumulated on the tips of T cell protrusions and, additionally, they showed that these peripheral protrusions are not only enriched in TCR components but also with the corresponding signaling molecules, such as Zap70, suggesting active signaling occurring in protrusions  to allow the TCR to bind to the pMHC molecule on the APC, and have been seen to localize to the tips of microvilli (Jung et al., The interplay between spatial organization of membrane proteins and membrane topological features, i.e., the characteristic heterogeneities of the T cell membrane, has functional consequences and significantly contributes to the fate of T cells. In this review we have covered research which demonstrates that the membrane has heterogeneous processes which affect T cell function. The dimensions of the morphological features which give the membrane biophysical properties, consequently allowing pre-clustering of proteins on the membrane surface, directly affect the organization of membrane proteins at the T cell/ APC contact interface, and though the study of these “initial” protein distributions is currently an active field of research, there remains the critical question of how these clusters are functionally advantageous and to what extent they increase the selectivity and sensitivity of the T cell. Reorganization of membrane proteins at the immunological synapse comes as a result of ligand-induced clustering, which in turn shapes the membrane itself. Thus, the reciprocal relationship between these classifications of T cell heterogeneity—protein pre-clustering and ligand-induced protein clustering—is that protein pre-clustering, due to biophysical mechanisms which determine the membrane shape, defines the protein distribution at the T cell/APC contact interface, which in turn shapes the membrane itself. We predict that this will continue to be an active area of research as more evidence is discovered to demonstrate the intrinsic relationship between membrane heterogeneity and T cell antigen recognition and subsequent activation.Although there have been huge advancements in investigating the effect of these membrane heterogeneities on T cell activation in recent years, there are still essential questions, to which, answers are yet to be discovered. Reports of the length of microvilli seem to range from 0.3 to 5 μm  but is tMF and SW wrote the manuscript and created the figures. KG and JG provided guidance and edited the manuscript. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


This study addressed the topology optimization of a carbon fiber reinforced plastic (CFRP) laminated battery-hanging structure of an electric vehicle. To accommodate parameterization for thickness and orientation of CFRP materials, the discrete material and thickness optimization (DMTO) technique was adopted. To include metal material as a reinforcement structure into the optimization simultaneously, the DMTO technique was extended here to achieve concurrent optimization of CFRP thickness topology, CFRP orientation selection and the topology of the metal reinforcement plate. Manufacturing constraints were applied, including suppressing intermediate void across the thickness direction of the laminate, contiguity constraint and the symmetrical layers. Sensitivities of the objective function were derived with respect to design variables. To calculate analytical sensitivities, finite element analysis was conducted and strain vectors were exported from a commercial software (ABAQUS) into a mathematical analysis tool (MATLAB). The design objective was to minimize the local displacement subject to the constraints of manufacturing and mass fraction. The mechanical performance of the optimized CFRP structure was compared with the original steel structure. To validate the optimization results, a prototype of the CFRP battery-hanging structure was fabricated and experimental testing was conducted. The results show that the total mass of the CFRP battery-hanging structure was reduced by 34.3% when compared with the steel one, while the mechanical property was improved by 25.3%. The transportation sector of the modern world mainly relies on fossil fuel. The use large amounts of fossil fuel is responsible for global warming, air pollution and ozone layer depletion. Besides, excessive usage of fossil fuel in vehicles is causing underground petroleum resources to dwindle. Environmental protection, lowering of tailpipe emissions and energy efficiency are among the most important goals of traffic policies, as transportation contributes to nearly one-third of greenhouse gas emissions ,2. In thIn fact, there are many studies underway to replace metal components with CFRP materials in vehicle structures. Wu et al.  addresseOptimization of CFRP structures is a good way to maximize the advantages of composite materials. Optimization of CFRP materials mainly includes two aspects, i.e., orientation optimization and thickness optimization. Structural performance may be completely different if different fiber orientations are selected due to anisotropic properties of CFRP materials . ActuallTo solve the problems of computational burden associated with heuristic optimization, various gradient-based topology optimization methods have been proposed to optimize FRP composite structures, such as the shape functions with penalization (SFP) method proposed by Bruyneel , the bi-In automotive industry, CFRP structures need to be connected with other components. Therefore, numerous studies have been carried out for analysis of CFRP-to-metal joints ,23,24,25The remainder of this paper is organized as follows: The DMTO method is first depicted in In this study, discrete material topology optimization (DMTO) approach was employed to parameterize different ply orientations  as continuous ones due to the requirement of a gradient-based optimization method, especially for a density-based topology optimization approach . Howevern scheme  is emplo imposed :(5)∑c=1Nlth layer in the eth element.In the conventional DMTO method, different candidate materials  is the normalized coordinate system in the thickness direction:In order to prevent the intermediate void, a Heaviside-like function is employed here ,31:(7)ρeBy introducing this function, the following relation in element The second manufacturing constraint is the contiguity constraint. It aims to avoid too many adjacent layers converging to the same orientation. When the total number of adjacent layers with same orientation is too large, there is a high probability of matrix cracking, so imposing the contiguity constraint is highly necessary. In this paper, the largest number of adjacent laminate layers with same orientation was set to be 2 as follows:The last manufacturing constraint is the symmetrical layers. Nowadays, there are many ways to manufacture CFRP composite structures, such as vacuum assisted resin transfer molding (VARTM), prepregs, heat-pressing forming and so on. Some techniques may result in undesired warping and deformation due to the residual stress during manufacturing. In this case, the constraint of symmetrical layers should be applied during the optimization process to avoid failed structures. This type of constraint can be illustrated asMesh-dependence and checkerboard patterns are the common issues in classic density-based topology optimization . In ordeIn this work, the density-based filtering technique was adopted for the design variables to avoid the checkerboard patterns after topology optimization . For varFor the gradient-based DMTO method, sensitivity analysis is essential because gradient information needs to be provided to the mathematical programming algorithm so as to solve for an optimum solution. If The sensitivity of thickness variable rived as (16)∂ρilIf the thickness variable t method  as(18)∂The battery-hanging structures on the floor are key components for electric vehicles because they are used to connect the floor and the battery pack. It is well known that the battery pack is of high weight and usually subjected to harsh environmental conditions, so the battery-hanging structure must have high strength and stiffness, with lighter weight being preferred. Therefore, CFRP material was adopted here due to its superior performance. The finite element (FE) model for the CFRP battery-hanging beam of an electric vehicle was developed in the commercial code ABAQUS. The equivalent single element ,35 was aIn this model, there are several hanging beams under the floor; after distributing the weight of the battery pack to each hanging beam, the average static load applied on a single hanging beam is 200 N. When 5 times the gravity acceleration is taken as the safety factor of the battery-hanging beam, then 6 times the gravitational force should be applied to the bolt in the center of the battery-hanging beam, so 1200 N force will be applied in the vertical direction. In fact, the floor does not bear the main load; instead, the floor girder and battery-hanging beam are the main load-bearing structures. The hanging beam mainly relies on the connection with the floor girder to transmit loads. Therefore, both sides of the battery-hanging beam are fully clamped in the finite element model, as shown in Four candidate ply orientations  were selected for the optimization problem for easy fabrication of CFRP. The battery-hanging beam is the connection structure between the floor and battery packs, and they are usually connected by bolts. In order to prevent the composite battery-hanging beam from failing at the position of the bolts, another choice of metal reinforcement plate was provided as a candidate material on the outermost layer of the structure, i.e., the location between the screw nut and the laminated battery-hanging beam. This section describes the simultaneous topology optimization of the metal plate and CFRP beam. Moreover, thickness topology optimization and orientation optimization of CFRP material were also conducted at the same time. Apart from the four CFRP orientations on the outermost layer, the metal plate was considered as the fifth candidate material. The material parameters for CFRP composites and metal (Q235) are summarized in In the initial design, the battery-hanging structure comprised 11 CFPR layers, and the metal plate was also able to be selected as a candidate material in layer 12. Therefore, there were five candidates in total to be selected for layer 12, i.e., ply orientations of −45°, 0°, 45° and 90° CFRP and metal material. A value of 0.25 was assigned for initial orientation variables from layer 1 to layer 11. In layer 12, the initial value of orientation variables was 0.2 because metal material was also regarded as a candidate material. The initial values for all of the thickness variables were set to be 0.5.η is the mass fraction which is set to be 50% of the mass In the definition of the optimization problem, the objective was to minimize the local displacement at the loading point of the battery-hanging beam while reducing the structural mass to meet the weight requirement. Accordingly, the design problem can be formulated asThe method of moving asymptotes (MMA) ,38 was eThe iteration history for topology optimization of the battery-hanging structure composed of CFRP and metal material is plotted in The layout of each layer of the battery-hanging beam after the topology optimization is shown in The thickness of the original Q235 steel battery-hanging structure is 1.2 mm and the total mass is 0.610 kg. To compare the optimized structure with the original steel one, the displacement diagram of the steel structure is shown in The optimized structure after postprocessing needed to be validated to determine the appropriateness and precision of the design results. Therefore, manufacturing and experimental testing were conducted, and the displacement at the location of load obtained from the test was compared with the deformation in finite element analysis. Prepregs made of unidirectional fibers were used for manufacturing. These pre-impregnated carbon fibers in which the epoxy resin already existed needed to be processed with the hot pressing technique to produce the CFRP prototype. The carbon fiber was T700 UD/epoxy resin prepreg, and the concave die and punch die were made of steel.Firstly, all possible contaminants on the surfaces of the molds were wiped with chemical cleaner. The release agent was applied on the molds to prevent the CFRP laminates from sticking to the surfaces. CFRP laminates were laid on the molds layer by layer, and the sequence of the layer orientations was according to the optimization results, as shown in To test whether the finite model for the optimal design was correct and accurate, the experimental tests of the prototyped battery-hanging structure were conducted, as shown in This study extended the DMTO method to conduct topology optimization of a carbon fiber reinforced plastic laminated battery-hanging structure together with a metal reinforcement plate. Considering the actual manufacturing process, the constraints of intermediate void, largest number of adjacent laminate layers with same orientation and symmetrical layers were applied in the topology optimization. The performance of the optimized structure was compared with the original steel counterpart, revealing that a reduction of 34.3% of the total mass for the CFRP battery-hanging structure was achieved without sacrificing mechanical performance. Specifically, the maximum local displacement was reduced by 25.3% when compared with the original steel structure, which implies that the stiffness was improved significantly with less weight of the total structure. Finally, the optimal design was prototyped and tested experimentally to validate the effectiveness of the proposed approach.


Radar measurements of gravitational mass-movements like snow avalanches have become increasingly important for scientific flow observations, real-time detection and monitoring. Independence of visibility is a main advantage for rapid and reliable detection of those events, and achievable high-resolution imaging proves invaluable for scientific measurements of the complete flow evolution. Existing radar systems are made for either detection with low-resolution or they are large devices and permanently installed at test-sites. We present mGEODAR, a mobile FMCW (frequency modulated continuous wave) radar system for high-resolution measurements and low-resolution gravitational mass-movement detection and monitoring purposes due to a versatile frequency generation scheme. We optimize the performance of different frequency settings with loop cable measurements and show the freespace range sensitivity with data of a car as moving point source. About 15 dB signal-to-noise ratio is achieved for the cable test and about 5 dB or 10 dB for the car in detection and research mode, respectively. By combining continuous recording in the low resolution detection mode with real-time triggering of the high resolution research mode, we expect that mGEODAR enables autonomous measurement campaigns for infrastructure safety and mass-movement research purposes in rapid response to changing weather and snow conditions. Gravitational mass-movements are a major hazard to infrastructure, inhabitants and tourists in mountainous regions. For snow avalanches in particular, authorities deal with the hazard by temporary or technical mitigation measures, for example timed closures of roads and ski resorts or the protection of infrastructure with long term mitigation measures like dams and galleries. However, real-time detection of avalanches is advantageous as it allows safety measures to be implemented only when needed, for example closing a road using traffic lights when mass movements like snow avalanches are detected on the mountain flank above. Currently, such real-time detection uses seismic or infrasound sensors in close proximity to the avalanche flow or radar technology that can observe the avalanche path from a stand-off distance.The first avalanche radars were of continuous wave type that measured speed utilizing the Doppler effect and a high spatial resolution was achieved by pointing a narrow beam to certain locations of the slope . Modern measured . While sTo increase the range resolution, a second generation of gravitational mass-movement radars were developed based on the FMCW (frequency-modulated continuous wave) principle. The first FMCW radar for snow movement was developed in 2010 and initially tested on a small-scale snow chute , and latstimated .GEODAR has been constantly modified and improved in the following years . A majorIn this paper, we present a portable FMCW radar which works in a similar fashion to GEODAR mark IV but consists of only one receiver channel. The main idea is to keep the radar compact and reconfigurable so that measurement campaigns can be performed quickly in response to weather and snow conditions at various locations. Therefore the radar must not only be portable but also requires a battery-based power supply. Furthermore, the radar needs the possibility to trigger a measurement on its own, since an external signal of the avalanche start is usually not provided. This leads to the development of a low-resolution detection mode to perform continuous data recording and real-time detection processing.Unfortunately, no data of a real mass-movement event could be acquired throughout the first year of operation, but we demonstrate the radar performance with freespace range data from a moving car. Additional loop cable test measurements show that the flexible high-frequency design of the hardware enables users to optimally choose settings of the frequency generation. We maximize the signal strength of the chirp pulses for the high-resolution research mode and low-resolution detection mode. We present a simple optimization method that utilizes only the hardware components and acquisition system itself and does not rely on any costly and sensitive high-frequency laboratory equipment. Thus, the signal generation settings can be optimized directly in the field.The radar hardware design and the signal generation together with the FMCW data processing are presented in The mGEODAR (mobile GEODAR) is an X-band FMCW-type radar with a centre carrier frequency of An overview of the mGEODAR hardware is shown in t-images . A band-The power amplifier (AM31-10.2-10.7-37-37) with 37 dB gain is in between the transmitter board and the transmit antenna. To keep the radiation pattern focused towards the avalanche track, the antenna specifications are chosen with a narrow azimuth angle and wider elevation angle. Two sector antennas (SA15-90-104V-D1 from Cobham) with 15 dB gain are used for transmit and receive purpose. Their beamwidth is around 85On the receiver board, the received signal is first conditioned with a series of low-noise amplifiers and a cascade of high- and low-pass filters. Then the signal is down-converted with the local oscillator frequency The resulting low-frequency deramp signal is passed through a custom built variable gain analogue baseband filter. The baseband filter characteristics are described in ocessing  is execuB and the pulse duration r is then a linear relation between this chirp rate, the signal propagation velocity c and the frequency f of the deramp signalA linear pulse in an FMCW radar is given by the bandwidth B by c with the speed of light gives The range resolution The maximum unambiguous range for an FMCW radar depends mainly on the largest resolvable frequency of the analog-to-digital converter (ADC), e.g., half the sample rate B and v of the target. The target should not move between two range gates during a single pulse, that is, the maximal velocity should by smaller then v causes a Doppler shift proportional to the speed and wavelength A further constraint on the pulse parameters earities . AdditioEquation . A targeThe wavelength s 0.03m 0 GHz andThe mGEODAR system comprises of two oscillators and a DDS as signal generation components that can be configured individually to give a variable range of output signals , the slow-time  and the chirp type (up- and down-chirp). Ash et al.  covers tThe slow-time processing, also known as moving target identification (MTI), compares the response of several consecutive chirp pulses. A high-pass filter along the slow-time axis for each range bin enhances changes between the pulses while suppressing static regions. The exact filter properties depend on the target, for avalanches usually a large passband is found optimal . HoweverThe radar performance and frequency configurations are tested with a coaxial cable loop setup connected between the transmitter and receiver board. The deramped signal intensity of a 90 To find optimal settings for the chirp signal The measurement for signal strength optimizations were done with a 90 Equation  for two-sinc-function, thus we expect decrease of signal with an increase of the chirp frequency The most limiting factor for the usable signal is the filtering of the chirp signal A which sency fch .We observe some spurious signals in the deramp signal. The instantaneous DDS frequency of these spurious is exactly a third of the set We optimize the signal intensities of both measuring modes with the loop cable. For each P at the apparent cable length  is of similar importance. We define the SNR as the ratio between the cable intensity f is very similar in both plots as 11 periods are clearly visible and the signal strength is comparable with around The deramp signal and the corresponding spectrum for the optimal frequency setting The range resolution for both measuring modes manifests in the spacing of the points in the frequency spectrum. For the detection mode, the peak of the cable measurement spreads across neighbouring value to about 10 A freespace range radar measurement was performed along a straight road with a moving car as target. Both radar antennas were set up approximately The colour intensity is a moving target identification (MTI) based on the digital filter described in The temporal and spatial resolution of both measuring modes is shown in the insets of Panel A and B that both span approximately over 20 The effect of the active baseband filter  is clearWide bandwidth radars for high-resolution gravitational mass-movement monitoring like the mGEODAR, and also its predecessor EREDAR, are challenging to construct as matching of wideband components and in-band spurious signals limit their performance. For example, the optimal frequency settings for the mGEODAR radar are slightly different from what to expect from the filter responses in Usually optimizations of high-frequency systems require costly laboratory test equipment. However, due to the integrated design of the radar boards, they cannot be used in intermediate stages between most of the components. Thus, we directly used the capabilities of the radar itself to find frequency settings that optimize the received signal strength. This approach has the advantage of including the whole signal chain in the optimization process.A similar optimization should be performed for each installation locality as for example the antenna cross-talk depends on the installation site and its geometry. Instead of a loop cable, such measurements could be done with active targets at several distances in the radar field of view. By knowing the geometric arrangement of the radar installation, potential cable lengths and the location of the active target with GPS measurements, the signal strength optimization procedure can done before any experiment. An active target may simply consist of two antennas with an amplifying component in between . These tsinc-function although gain flatness is desired [For the current mGEODAR system we find an optimal LO frequency setting for the 400  desired . We postAnother possibility to increase the gain flatness would be to change the clock frequency y tested , the chiThe current radar system can still improve on the pulse coherency in the research mode as the standard deviation between 100 pulses is relative high A. As a rWhen this trigger impulse is generated by a digital output of the acquisition system, the binary marker that indicates the beginning of a pulse does not need to be captured. Instead, the acquisition system can be set to record only the deramp signal channel and the full recording speed of 2 Up until now the data processing is performed offline and no trigger conditions to switch between the detection mode and research mode have been identified yet. The next step is to acquire real avalanche data, develop real-time data processing and motion detection algorithms. Since direct velocity estimation with FMCW technique is complicated and so far rather impossible for large distributed targets that constantly change the reflectance due to variable composition and deformable surface, any motion detection needs to compare the return signal between several pulses, thus, apply image identification and classification algorithms to the 2-dimensional range-time MTI data .The signal intensity for the moving car is difficult to compare with the signal strength expected from an avalanche. The car is a point target and its scattering intensity is usually higher than the background, in contrast for avalanches, the background scattering of the snow cover is of similar magnitude to the moving snow. However, the previous FMCW radars GEODAR I to V have indeed enough sensitivity to successfully image very different avalanches that range from small loose snow avalanche to large powder snow avalanche and from dry to wet snow avalanches . Still, We present the hardware of a custom-build mobile FMCW radar system for the purpose of detecting, monitoring and measuring gravitationally driven geophysical mass-movements such as snow avalanches. We anticipate that very similar radars will be employed in the future to monitor other natural mass movements like debris flow, rock fall and also very rare pyroclastic density currents on volcano flanks. However, the development of such mass-movement detection radars definitively benefits from snow avalanche research as these flows can be triggered manually and occur comparably more frequent throughout the complete winter season. Thus, the hardware, software settings and real-time processing algorithms can be developed more easily for snow avalanches and later adopted to match the other mentioned flow types that are sporadic natural events and usually exhibit a complex transient signature in time and space. Indeed, more research is required in feature recognition algorithms on the range-time data to differentiate a mass movement event from other moving targets such as waving trees in the field of view.For various measurement locations in rapid response to changing climate and snow conditions, the presented optimization procedure will prove useful to quickly find the optimal frequency settings for any new measurement location. We see a main advantage of the procedure, that basically sweeps through the full valid frequencies of the hardware, without the need for additional sensitive laboratory equipment. Employing solely the radar hardware and capability of the analog-digital converter ensures that the complete signal chain, the cabling and antenna orientations as well as potential external noise sources are optimally taken into account at each location.Even though no natural avalanches could be recorded until now, the presented tests and results of a non-distributed moving target are promising, however, further evaluation on real events is needed. In the future, the data acquisition and processing will be further optimized for computational speed and efficiency. A next step will be the development of a hardware-based FMCW processing using FPGA technology rather than software-based offline processing which requires decent computing power and energy consumption. Beside the flexible usage cases for the radar, the overall costs of all mGEODAR hardware components are around 10 k Euro and will promote the wide use of radar technology in mass-movement and general environmental monitoring, detection and measurements across various disciplines and as ground-truthing for satellite based radar remote sensing of, for example, the snow cover properties.


In the underlay cognitive radio networks, the radio environment maps (REMs) estimation is the main challenge in sensing the idle wireless spectrum resources. Traditional deep learning-based algorithms estimate the REMs on the basis of the high-quality, large-scale complete training images. However, collecting the complete radio environment images is time-consuming and requires a numerous number of power spectrum sensing nodes. For this reason, we propose a generative adversarial networks-based pixel regression framework (PRF) for underlay cognitive radio networks. The PRF algorithm relaxes the requirement of the complete training images, and estimates the radio environment maps only on the basis of the incomplete REMs images, which are easier to be collected. First, we transform the radio environment maps estimation task into a pixel regression task through the color mapping progress. Then, to extract helpful information from the incomplete training data, we design a feature enhancing module for the PRF algorithm, which intelligently learns and emphasizes the important features from the training images. Finally, we use the trained pixel regression framework to reconstruct the radio environment maps in the target area. The proposed algorithm learns accurate radio environment characteristics from the incomplete training data rather than making direct biased or imprecise radio propagation assumptions as in the traditional methods. Thus, the PRF algorithm has a better REMs reconstruction performance than the traditional methods, as verified by simulations. Cognitive radio network (CRN) is a promising technology to optimize the utilization of the spectrum resources . In the There are two types of users in the cognitive radio network: PUs (primary users) and SUs (secondary users) . The priAn important radio resources allocation scheme is the underlay cognitive radio network, which is also named as the spatial reuse CRN ,7. In thIn the underlay cognitive radio network, we adopt the radio environment maps (REMs) to display the primary users’ power spectrum (PS) in the CRN area. It is a visible map of the wireless environment, which adds the PS information into the spatial map. Based on the estimation of the REMs, we can handle the conflict between the SUs and PUs and maximize the utilization of the spectrum resources in a particular CRN region.Estimating and utilizing the radio environment maps is extraordinarily helpful in wide-area CRN . To reduAs shown in In fact, the above goal is undetermined. There are infinite REMs which can satisfy the above known SUs’ parameters. To reduce the solution space, we should utilize the prior information of the radio environment in the target region. However, traditional methods ,13,14,15As a significant method of the artificial intelligence (AI), deep learning technology has irreplaceable advantages in the radio environment maps estimation task . By explIn our previous work , we propAs shown in Compared with the complete REMs, the incomplete REMs are easier to be collected. We can simply collect them in advance by setting a small amount of power spectrum sensing devices in the target region. Training the deep neural network to learn from the incomplete REMs training data is a more realistic way, which has not been reported until now.To estimate the REMs for underlay CRNs, we propose a GAN-based PRF algorithm using only incomplete REMs training data. Using the incomplete training data to estimate the radio environment maps in the target region has not been reported until now.In the MEGANs algorithm, the discriminator is trained to distinguish if the input complete images are from the real complete training data set or from the REMs estimated by the generator. However, unlike the MEGANs settings, we only have a dataset of incomplete REMs training images. In this paper, we propose a pixel regression framework in which the discriminator is trained to identify if the input incomplete images are sampled from the real complete REMs or from the REMs estimated by the generator.Compared with the complete training data, the incomplete REMs images contain relatively less information. To improve the feature extraction ability of the neural network in MEGANs, we design a feature enhancing module in our PRF algorithm. The proposed module can extract more information from the incomplete training data and intelligently emphasize the important features of the radio environment maps.In this paper, we propose a generative adversarial network (GAN) -based REMs estimation algorithm named pixel regression framework (PRF) for underlay CRNs. Compared with our previously proposed MEGANs , the PRFThe rest of our paper is organized as follows. In The acronyms in this paper are shown in The radio environment map is a powerful tool to determine the PUs’ signals across a finite geographical area. On the basis of the REMs, we can obtain the distribution of the signal strength in the target region. As for the applications of the radio environment maps, DARPA developed its advanced radio frequency mapping program, i.e., Radio Map, to achieve the real-time sensing of the spectrum resources across space and frequency domains . Based oTo solve the undetermined REMs estimation problem, initial efforts have been made by utilizing the prior information or assumptions of the radio environment in the target area. Conventional radio environment maps estimation methods include spatial interpolation methods ,14,15,20The spatial interpolation methods include the inverse distance weighted (IDW) interpolation algorithm and the Kriging spatial interpolation algorithm. The inverse distance weighted interpolation supposes that the PS only depends on the distance ocations ,15. The As for the Kriging interpolation method, it estimates the radio environment maps with the weighted additions of the known PS parameters on the basis of the semi-variogram functions. We can regard the above functions as the latent features of the wireless environment in the target area ,13, whicAll the above algorithms perform well in some simple environments. However, the real wireless environment is quite complicated. In the practical radio environment, the signal is attenuated in a random fashion. The attenuation is mainly caused by the radio propagation loss, the shadow fading, the multi-path effect, etc. Under the superposition of the above attenuation styles, building a high-accuracy mathematical model for the radio environment is quite difficult. However, all the above traditional REMs estimation methods make direct assumption about the radio environment. Biased or inappropriate hypotheses will lead to inaccurate radio environment maps estimation results. For example, the IDW interpolation algorithm assumes that the power spectrum only depends on the distance between the PUs and SUs ,15. The Deep learning is a promising technology to estimate the radio environment maps. On the basis of the deep neural networks, we can obtain the prior knowledge of the wireless environment by extracting the latent propagation features from the training data set, which makes the REMs estimation results more intelligent and precise. However, there are few studies which focus on deep learning-based REMs estimation algorithms .G) and the discriminator (D) [In our previous work , we propator (D) . The strator (D) . The genIn the MEGANs, the generator and the discriminator are trained on the basis of the complete training data. However, collecting the complete REMs is time-consuming and requires a numerous number of sensing nodes. In this paper, we propose a novel GAN-based REMs estimation algorithm named PRF algorithm. The proposed method can extract helpful information from the incomplete REMs training data, which are relatively easy to be collected.T. The SUs and PUs are located at We assume that there are We use Equation , we adopWe suppose that In fact, the above goal is undetermined. There are infinite REMs which can satisfy the constraints in Equation . To compIn the color mapping process, we divide the target region into On the basis of the color mapping process, we transform the REMs estimation task into a pixel regression task. Then, the efficient regression method—generative adversarial networks can be used to solve the REMs estimation problem.In our previous work , we propOn the basis of the MEGANs, we propose an improved GANs-based algorithm named pixel regression framework, as shown in r" denotes the true or real power spectrum distribution in the target area. Superscript "e" denotes the estimated or generated PS distribution from the generator. We use "C" to denote complete REMs images and "I" for incomplete REMs images. For example, Throughout, the superscript "G), a discriminator (D) and a sampler In our proposed algorithm, the discriminator is trained to identify if the input incomplete images are sampled from the real complete REMs (2)If we force It should be noted that we do not use the same sampling locations as the true incomplete REMs images, i.e., the color grids’ locations of On the basis of the MEGANs , the impT.Regarding the generator, the proposed structure is designed in the light of the auto-encoders . The genAs for the discriminator of PRF, a deep convolutional neural structure is utilized to distinguish the true incomplete REMs from the estimated incomplete REMs . In addiIn MEGANs, the deep neural network is trained to extract useful information from the complete training images. However, compared with the complete training data, the incomplete REMs images contain relatively less information. To improve the feature extraction ability of the neural networks in MEGANs, we design a feature enhancing module in our PRF algorithm, as indicated by the red modules in To extract the latent radio environment characteristics from the incomplete training images, we need to improve the feature extraction ability of the neural network. Adding more layers to the original neural network is a general method , but deeOn the basis of the squeeze-and-excitation block , we desiRegarding the FE sub-module, it only includes two convolutional layers, which deepen the neural network and extract more information from the incomplete training images. It should be noted that the two convolutional layers employ We assume that the input of the FE sub-module is As for the FW sub-module, it weights and emphasizes the important features and the detail features. The FW sub-module includes 3 main processes: the image entropy pooling, the max pooling and the fully connected bottleneck layers.Regarding the image entropy pooling, we compute the channel-wise image entropy of the input image, which measures the information of different channels. The image entropy pooling process extracts the detail features of different feature maps. The fully connected bottleneck layers are trained to emphasize the feature maps on the basis of the above pooling results.i th channel’s image entropy of Y. We assume that the image entropy pooling function is We use Equation .(3)E=fIAs for the max pooling process, we compute the channel-wise maximum value of the input image, which measures the prominent information of different channels. The max pooling process extracts the importance of different feature maps. The fully connected bottleneck layers are trained to emphasize the feature maps on the basis of the pooling results.i th channel’s maximum value of Y. The max pooling function is We use Equation .(4)M=fMRegarding the fully connected bottleneck layers, they set different feature maps with different channel weights, which can be regarded as a self-attention function towards different channels.i th channel of Y and its weight is The feature weighting process is shown in Equation  and EquaIn order to deepen the neural network and avoid increasing training difficulty at the same time, we adopt the widely used residual mechanism in the residual networks . ThroughIn this section, on the basis of the feature enhancing process and the residual mechanism, we emphasize the important features and avoid increasing training difficulties at the same time. In addition, the feature enhancing module maintains the dimensions of the inputs and the outputs of the original feature maps. The dimension maintaining ability makes the proposed feature enhancing module a more generic extension, which can be added to the original neural network.d is the distance between PU and SU. The preselected constants, In the practical wireless radio environment, the large scale fading includes two factors: the radio propagation loss and the shadow fading. In the simulations, we adopt the inverse polynomial law model ss model . γpr is T into T. We assume that two transmitting primary users are located at grids  and , under the AWGN with known variance T. The number of SUs is about 15% of all We divide the testing area As for the training data set, we use the same radio environment model but different parameters. We generate 20,000 training images from two sets of propagation parameters: (1) D and G are 0.0004 and 0.0001; the batch size is 24; the gradient penalty coefficient is 10 [In the PRF training process, the learning parameters are as follows: we use Adam algorithm and the learning rates of nt is 10 .ith real and estimated REMs in every training batch. To monitor the estimation ability of the generator in the training process, we define the Euclid distance Equation . Mri andWe use PRF-NFE to denote the PRF without the feature enhancing modules. As shown in In addition, although PRF converges faster than PRF-NFE from the perspective of the number of iterations, the PRF needs more time than PRF-NFE per iteration. On the basis of the Intel Core i7-8750H processor and RTX 2060 graphics card, the PRF and PRF-NFE need 6.7 s and 2.5 s per iteration, respectively. We prefer the PRF algorithm for 3 reasons: (1) the PRF algorithm has a better estimation performance than PRF-NFE, as verified by the following subsection; (2) The training data are limited in most cases. As for the REMs estimation task, a neural network with stronger information extracting ability is important because of the limited and incomplete training images; (3) A computer with a more powerful calculation ability can solve the time-consuming problem for the training process of the PRF algorithm.We compare the proposed PRF with the PRF-NFE, IDW and Kriging algorithm on testing data. We use the the Kriging with the exponential semi-variogram. As for the IDW algorithm, the power value of the inverse distance is set to be We select three indicators to test the pixel regression framework: (1) The visual display of the radio environment maps; (2) the estimated power spectrum of primary users; (3) the average REMs estimating error , which demonstrates that the proposed module enhances the approximation ability and the feature extracting ability of the pixel regression framework.Regarding the performance of the estimated power spectrum, we compare the PRF reconstruction results with the true primary users’ power spectrum in The proposed PRF algorithm outperforms the PRF-NFE because of the feature enhancing module, which enhances the approximation ability and the feature extracting ability during the same training epochs.Regarding the Kriging interpolation algorithm in As for the IDW algorithm, the influences of the available nodes on the estimated nodes are controlled by the power value of the inverse distance, i.e., f th frequency point. The radio environment maps are reconstructed on the basis of the power spectrum from the receiving users. The more power spectrum from receiving users, the better estimated performance from the pixel regression framework. Thus, the As for the average REMs estimating error (Equation .(9)AREEWe choose the estimated REMs randomly at 20 frequency points and calculate The average REMs estimating errors of the PRF and the PRF-NFE decrease with the increase of the number of secondary users gradually in Compared with the PRF-NFE, the PRF performs better because of the feature enhancing module, which helps the neural network to extract more information from the incomplete training data. In addition, the reconstruction performance of PRF is not good enough at 10%. It is mainly caused by the fact that the amounts of the PS measurements are too small to activate the neural network of the PRF in the data forward propagation process. To solve this problem, we can select a smaller value to the parameters of the measurement function in the PRF training process, e.g., Regarding the IDW method, it performs worse compared with PRF. The inaccurate reconstruction result is from the imprecise power value setting in IDW, which controls the influence of the receiving users on the interpolation points.As for the In this paper, we propose a generative adversarial networks-based radio environment maps estimation algorithm named pixel regression framework. The proposed PRF algorithm relaxes the requirement of the complete training images in the traditional deep learning-based method and can estimate the REMs using only incomplete REMs training data. To improve the feature extraction ability for the incomplete REMs, we design a feature enhancing module for the PRF algorithm, which intelligently learns and emphasizes the important features from the REMs images. We simulate a radio environment with the radio propagation loss and the shadow fading, and select three indicators to test the proposed algorithm: the visual display of the radio environment maps, the estimated power spectrum of PUs, and the average REMs estimating error against different numbers of secondary users. Compared with the traditional methods, the PRF algorithm performs better upon the above indicators in the simulations. In our future research, we will concentrate on utilizing the correlation information between adjacent frequencies’ training images to improve the REMs estimation performance.


In this paper, we propose a novel emotion recognition method based on the underlying emotional characteristics extracted from a conditional adversarial auto-encoder (CAAE), in which both acoustic and lexical features are used as inputs. The acoustic features are generated by calculating statistical functionals of low-level descriptors and by a deep neural network (DNN). These acoustic features are concatenated with three types of lexical features extracted from the text, which are a sparse representation, a distributed representation, and an affective lexicon-based dimensions. Two-dimensional latent representations similar to vectors in the valence-arousal space are obtained by a CAAE, which can be directly mapped into the emotional classes without the need for a sophisticated classifier. In contrast to the previous attempt to a CAAE using only acoustic features, the proposed approach could enhance the performance of the emotion recognition because combined acoustic and lexical features provide enough discriminant power. Experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpus showed that our method outperformed the previously reported best results on the same corpus, achieving 76.72% in the unweighted average recall. Emotions play an important role in successful communication among humans , and thuOne of the main research topics for speech emotion recognition is to find appropriate features that discriminate different emotions ,31,32,33Recently, deep learning approaches have been incorporated in speech emotion recognition and brought performance improvement ,31,32,33In this paper, we propose a speech emotion recognition system that captures emotion-related underlying factors using a CAAE  from acoOne of the popular models for emotion in psychology is the circumplex model , which rMany previous works ,16,17,18In this paper, we used the acoustic and lexical features similar to those in our previous work . As for The utterance-level acoustic features were derived from the frame-level features applying statistical functions . The uttG, decoder G, G trying to make the output ẑ obtained as an output of G given an additional condition G network was used to extract low-dimensional latent variables ẑ from the input feature vector.In the proposed system, a CAAE was adopted to map the combined utterance-level acoustic and lexical features to low-dimensional latent representation to discover underlying factors relevant to speech emotion classification. The architecture of the CAAE is shown in ẑ generated by the generator. The adversarial network then updated the generator to capture the prior distribution to confuse the discriminator. In the experiment, we selected the dimension of the latent variable, n, as two and used a four-mixture bivariate GMM as the prior distribution. Each mode of the GMM corresponded to one of the four emotion classes with equal probability, i.e., https://mspl.gist.ac.kr/ser_confusionmatrix/confusion_matrix.html).Training of the CAAE was carried out by alternating two phases, which were the reconstruction and regularization phases. In the reconstruction phase corresponding to the first term of the cost Equation , the autIt is noted that the configuration in Experiments were performed for the IEMOCAP dataset  to demonThe experiments were conducted in a leave-one-speaker-out cross-validation scheme to focus on speaker-independent emotion recognition. The test data for each of the 10-fold cross-validation session consisted of the utterances spoken by one specific speaker, while the training data became the utterances from the other 9 speakers. The performance was assessed using the weighted average recall (WAR) and the UAR. The WAR is the ratio of the total number of correctly predicted test samples and the total number of test utterances, while the UAR is defined as the accuracy per class averaged over all classes so that the accuracy for each class has the same importance regardless of the number of test samples in the class.The proposed CAAE model consisted of the encoder, generator, and discriminator, each of which had 2 hidden layers of 1024 rectified linear units . The actIn The performances of the proposed and previously reported emotion recognition systems ,23,25,28The distribution of the learned latent vectors extracted by a CAAE for the training and test sets with acoustic features are shown in The confusion matrices for the proposed method with a DNN classifier and that with a simple classifier checking signs of the components are shown in In this paper, we proposed a multimodal emotion recognition method that captured underlying emotional characteristics from the joint representation of acoustic and lexical features using a CAAE. The acoustic and lexical features were combined and fed into the CAAE to discover the underlying emotion-related low-dimensional representation, which may be interpreted in the VA space in the circumplex model. With a proper design of the prior distribution in the CAAE, the emotional class could be simply decided according to in which quadrant the latent two-dimensional vector fell. Our experiments on the IEMOCAP dataset showed that the proposed method outperformed all other methods that reported the performances for the same dataset in both WAR and UAR.


Dynamic motor imagery (dMI) is a motor imagery task associated with movements partially mimicking those mentally represented. As well as conventional motor imagery, dMI has been typically assessed by mental chronometry tasks. In this paper, an instrumented approach was proposed for quantifying the correspondence between upper and lower limb oscillatory movements performed on the spot during the dMI of walking vs. during actual walking. Magneto-inertial measurement units were used to measure limb swinging in three different groups: young adults, older adults and stroke patients. Participants were tested in four experimental conditions: (i) simple limb swinging; (ii) limb swinging while imagining to walk (dMI-task); (iii) mental chronometry task, without any movement (pure MI); (iv) actual level walking at comfortable speed. Limb swinging was characterized in terms of the angular velocity, frequency of oscillations and sinusoidal waveform. The dMI was effective at reproducing upper limb oscillations more similar to those occurring during walking for all the three groups, but some exceptions occurred for lower limbs. This finding could be related to the sensory feedback, stretch reflexes and ground reaction forces occurring for lower limbs and not for upper limbs during walking. In conclusion, the instrumented approach through wearable motion devices adds significant information to the current dMI approach, further supporting their applications in neurorehabilitation for monitoring imagery training protocols in patients with stroke. Motor imagery (MI) is the mental representation of an action without its physical execution  and it rIn real practice, such as sports training or rehabilitation, MI does not replace physical exercises and the importance of decoupling MI from the real execution of action decays, allowing the possibility of coupling them to enhance the efficacy of the training ,7. For eRecently it has been pointed out that dMI of a walking task may be a suitable tool for recovering locomotor functions in patients with gait disorders of an orthopedic nature . AdditioDespite few results reporting that stroke patients may have some deficits in terms of motor imagery , it has The use of dMI exercises may also allow the assessment of MI performance, not only from a temporal point of view, but also considering its spatial features. In fact, considering all the above considerations, it is of great importance to objectively verify whether people experiencing a stroke are also able to imagine complex coordinated actions such as the limb oscillations during locomotion. Limb oscillations could be modeled as those of a pendulum, in which the optimal frequency of oscillations depends on the limb length, but not on its weight, according to the physical law of isochronism . ApplyinMany studies have already investigated coordinated movements in stroke patients, often focusing alternately on the upper limbs  or on tWith respect to MI, the assessment of imagery performance during dMI could be easier, because the accompanied movement could be quantitatively assessed using wearable inertial devices ,35.The aim of the present study is to investigate the capacity of patients with stroke to imagine walking, mentally reproducing the actual movements they do during real walking, in particular the limb oscillations. For this purpose, the dMI allowed to measure, by wearable inertial sensors, the limb oscillations performed by patients mimicking in place those usually performed during walking. The comparisons of upper and lower limb oscillations during dMI with those performed in simple motor tasks without any imagery and with those performed during actual walking could allow to test motor imagery in patients. Their data were compared with those of age-matched healthy adults as controls and those of young adults as a physiological reference. Our hypothesis is that single limb swinging is poorly related to the swinging of the same limb during walking, when coordinated among all the four limbs, but this difference can be reduced by dMI, in terms of waveform, angular speed amplitude and frequency of movements.All measurements and experimental conditions have been performed in accordance with the Declaration of Helsinki standards. The study was approved by the Local Independent Ethics Committee of IRCCS Santa Lucia Foundation  (protocol number: CE/PROG.738). Every subject provided written informed consent prior to participation. We recruited both healthy subjects and patients with stroke at the Private Neurorehabilitation Unit of Santa Lucia Foundation.To investigate the physiological performances, the possible age-related decline, and pathological performances affected by stroke, three groups were enrolled, as in a previous study on locomotor imagery : a firstFor the YG and CG subjects, the inclusion criteria were: (1) be of legal age; (2) have Italian as their mother tongue or be resident in Italy for more than ten years; (3) not have sensory or motor deficits or orthopedic problems that can influence motor performance.3 wooden blocks [Patients were clinically evaluated through tests and scales commonly used for assessing the most frequent stroke deficits : gait abn blocks ). ParticThe experimental procedure is made up of four conditions  and to oscillate one limb in the sagittal plane in each trial. Four trials were performed (one for each limb) in a randomized order given by the researcher, and each trial lasted 30 s. During the oscillations of a lower limb, the subject was the on the step with their contralateral leg, whereas the oscillating leg is moved in the sagittal plane in the air. To avoid losing balance, participants required handles to lean on which were on a braked walker positioned in front of them with handles close to the subject’s hands.Subsequently, the dMIt was recorded. The tasks were the same of SW, but now subjects were also asked to imagine walking during limb swinging . No indication was given regarding maintaining open or closed eyes. In the CT condition, we asked the participants to position themselves in front of the first target (start) and to imagine with open eyes the time necessary to reach the second target (stop) placed on the straight path, without performing any movements. No information was given about the length of the path. The examiner synchronized the start of the stopwatch with a vocal start at which the participant began to imagine, the stopwatch was stopped when the participant verbally stated that he/she had mentally reached the second target ,38. The For recording the movements, each participant was equipped with four synchronized magneto-inertial measurement units (MIMUs) , placed on both ankles and wrists. The two MIMUs placed on the right and left wrists were positioned between the ulna bones and the radius, while those on the right and left ankles were positioned slightly above the lateral malleoli. The MIMUs were securely fixed to participants’ bodies using multiple elastic velcro straps that adjust to fit different body sizes. These fixing devices allowed the prevention of any possible device displacement from its initial location. The MIMUs were located on the anatomical district and then carefully aligned to better approximate the cranio-caudal, medio-lateral and antero-posterior axes of the body segments. To guarantee a repeatable system of reference for all participants, an extra sensor was located on the floor, with one axis aligned with the direction of progression and another axis parallel to the gravity vector. A 5 s trial was recorded at the beginning of each trial when the participant was asked to be firm maintaining standing position. This static acquisition allows for the computation of the orientation of the current local frame and then to rotate the local reference systems via software through a rigid transformation on the basis of the extra-sensor orientation, in order to obtain an anatomical reference frame defined by the sagittal plane and its orthogonal axis . Each MIp > 0.6). The curve obtained from the sinusoidal fit was analyzed, allowing for the characterization of swinging movements by the following three parameters: AV = a, the amplitude of the angular velocity of movement (rad/s); Fq = 1b/(2π), the movement frequency (Hz); the similarity to a sinusoidal waveform computed by using the coefficient of determination R-squared (R2) of the sinusoidal fit.The raw data processing was performed using the MATLAB software . The analyzed data referred to those obtained by gyroscopes along the sagittal plane. Data were pre-processed and digitally low-pass filtered, using a fourth order Butterworth filter with a cut-off frequency of 2 Hz. This cut-off frequency was chosen according to the Nyquist theorem, and based on above reported data related to limb oscillations and stride frequencies that are lower than 1 Hz in the human gait . As the According to the literature on the Imagery Performance Index (IPI) , the simt-tests as post-hoc comparisons. For data that was not normally distributed (R2 and IPIs), nonparametric statistics were used for inferential analyses. A Krukal-Wallis analysis was used to perform between group comparisons among the data of YG, CG and those of PG divided by the affected an unaffected side (in relationship to hemiparesis), followed by post-hoc analyses performed with Mann-Whitney u-tests corrected for Bonferroni. A Wilcoxon test was used to compare SW and dMIt IPIs within each group. The Spearman coefficient was used to assess the entity of possible correlations between the frequency of leg or arm oscillations and the root square of the ratio between the acceleration of gravity and leg or arm length. For all the tests, the significance level was set at 0.05, with the exception of post-hoc analyses  and correlations, for which the alpha level of significance was divided by the number of possible comparisons (12) and set at 0.00417, according to Bonferroni correction.The statistical analyses were performed using the version 23 of IBM SPSS statistical software. Data were summarized in terms of the median and interquartile range (third less first quartile) for the clinical scales, mean and standard deviation for continuous measured parameters, and the mean and standard error of the mean for the IPI percentages. The normal distribution of each parameter was verified using the Shapiro-Wilk test. For normally distributed data, an analysis of variance was performed followed by p = 0.096 t-test, gender: p = 0, chi squared test) and anthropometric (p > 0.05) characteristics. The clinical assessment of patients, which depicts the level of their deficits, is reported in As shown in p < 0.001), but not its frequency . Interestingly, a significant interaction was found also for condition * group * limb  for the speed amplitude. In fact, as shown in Performance data are reported in As shown in p = 0.003) and SW , but not in dMIt . Interestingly, in healthy subjects, the frequency of upper limb oscillations was found to be significantly correlated with the root square of ratio between gravity acceleration and arm length in CW  and dMIt , but not in SW . In CW, the frequency of upper limb oscillations was found significantly correlated with that of lower limbs . Conversely, for patients during actual walking, neither the frequency of oscillations for upper limbs  or for lower limbs  were significantly correlated to relevant anthropometric measures.In the YG and CG, the mean frequency of lower limb oscillations was found to be significantly correlated with the root mean square of ratio between gravity acceleration and leg length in CW  . The impIn patients, the deficits due to stroke may have greatly impaired the movements and reduced the capacity of exploiting the pendular mechanism of walking . It may Considering the overall results about healthy subjects and patients, at this point, an instrumented movement analysis during dynamic motor imagery seems to be important for objectively measuring the ability of patients to imagine the realization of the motor gesture and to compare it with the actual locomotor execution. The sensory feedback should be included in motor imagery training protocols, for improving the equivalence of dMI with actual gestures. In the future, rehabilitative programs could provide the application of biofeedback during motor imagery in order to improve patient’s performance during locomotion, especially for lower limbs. For upper limbs, this approach was already suggested using a brain–computer interface during motor imagery , whereasThe definition of MI, id est the mental representation of an action without its physical execution , seems tGiven the purpose of this study, MIMUs were fruitfully used for assessing the motor behaviours of healthy and stroke subjects. Previous studies used a single MIMU for comparing dMI and actual walking ,39, but So, this method of investigation allowed an objective and sensitive evaluation of how patients imagine their movements during mental representation of walking. Upper limbs seemed benefit more from dMI than lower ones, probably because the latter were moved in a more automatic way that also depends on sensory feedback, reflexes and ground reaction force. The importance of upper limbs oscillations was already reported in the scientific literature for a correct implementation of locomotor schemes .Taking up the concept of the famous neuropsychologist Luria about kinetic melodies , the movIn conclusion, our findings showed that dynamic motor imagery may positively influence locomotor movements both in healthy and pathologic people, more in upper than lower limbs. However, this study did not deal with a treatment based on dMI. Further studies should investigate whether the use of dMI protocols involving coordinated movements based on paired (in-phase and out-of-phase) oscillatory movements of multiple limbs together can help restore harmonious movements in stroke patients. In the light of our results, rehabilitation protocols based on dMI should include the contact with the ground when lower limb movements are mimicked and the measure of limb movements by wearable devices for monitoring the congruence of patients’ imaginations with the actual execution of movements.


It is also observed that the benefit of using an FoV-expanded camera array does not diminish for subjects who are more experienced. This test provides convincing evidence and validates the hypothesis that expanded FoV with small blind spots can reduce the operation time for laparoscopic surgical tasks.Existing laparoscopic surgery systems use a single laparoscope to visualize the surgical area with a limited field of view (FoV), necessitating maneuvering the laparoscope to search a target region. In some cases, the laparoscope needs to be moved from one surgical port to another one to detect target organs. These maneuvers would cause longer surgical time and degrade the efficiency of operation. We hypothesize that if an array of cameras can be deployed to provide a stitched video with an expanded FoV and small blind spots, the time required to perform multiple tasks at different sites can be significantly reduced. We developed a micro-camera array that can enlarge the FoV and reduce blind spots between the cameras by optimizing the angle of cameras. The video stream of this micro-camera array was designed to be processed in real-time to provide a stitched video with the expanded FoV. We mounted this micro-camera array to a Fundamentals of Laparoscopic Surgery (FLS) laparoscopic trainer box and designed an experiment to validate the hypothesis above. Surgeons, residents, and a medical student were recruited to perform a modified bean drop task, and the completion time was compared against that measured using a traditional single-camera laparoscope. It was observed that utilizing the micro-camera array, the completion time of the modified bean drop task was  Laparoscopic surgery involves the use of a thin, tubular device called a laparoscope that is inserted through a keyhole incision into the abdomen to perform operations that used to require the body cavity to be opened. Because the procedure involves smaller incisions, laparoscopic surgery promises less trauma, shorter hospital stays and faster recoveries . The lapLaparoscopy is a technically complicated surgery that requires excellent hand-eye coordination and an almost intuitive ability to navigate delicate internal structures. The surgeon must also rely on a patient-side assistant to position the camera correctly. Furthermore, the limited field of view (FoV) afforded by a single-camera laparoscope makes it difficult to navigate inside the body cavity to reach the targeted surgical region, or to monitor the surrounding anatomy as in an open-cavity surgery.Several improvements to the current design of the single-camera laparoscope have been proposed. Most of these proposals focus on replacing the single-camera assembly with a pair of binocular cameras to provide 3D vision ,3,4. Bhastortion , and mayA second direction for improving the laparoscope is to incorporate multiple (more than two) cameras to enlarge the FoV. The MARVEL system ,16,17 usTo alleviate the limited FoV and low efficiency of operation in traditional laparoscopic visualization, previously, we demonstrated a trocar-camera assembly (TCA) . HoweverChanging the camera array from a 75 mm arm length with 0This micro-camera array that uses tilted cameras provides larger FoV compared to that accomplished via non-tilted cameras c,d. For In this work, we hypothesize that if an array of cameras can be deployed to provide a stitched video with an expanded FoV and small blind spots, the time required to perform multiple operations at different spots can be significantly reduced. While this seems to be intuitively correct, to the best of our knowledge, no empirical study has been performed to validate this hypothesis.To validate this hypothesis, we conducted an experiment that compares the time taken to complete a laparoscopic surgical training task using a proof-of-concept micro-camera array against that of using a conventional single-camera laparoscope. We utilized a simplified version of the TCA which contained cameras with optimized tilting angles but without the deployment mechanism  21]. In. In21]. The micro-camera array visualization system consisted of a micro-camera array, a trainer box, video data transfer boards, operating monitor, and a laptop implementing a video stitching algorithm a. We useous work , the totra array d,e. Camera array d. The tiTo validate the effectiveness of this proposed micro-camera array on the laparoscopic surgery, we mounted it on to an FLS laparoscope trainer box (to be discussed a bit later) and compared the FoV of the stitched video frame against the FoV of a single-camera visualization system and a non-tilted camera array. For comparison, we showed the images of the same grid paper under these three visualization systems in In this work, we succeeded in achieving larger FoV with fewer blind spots than in the previous work that used non-tilted cameras . FurtherA real-time multi-view video stitching algorithm was developed to process video feeds from the five cameras to form a video mosaic whose FoV is the union of FoVs of all five cameras. At this early stage of development, we designated the non-tilted center camera as the default viewpoint (the main view) and augmented its FoV with the four peripheral cameras (the side views) using a stitching algorithm similar to the one proposed by Zheng et al. . At the For each video frame, the FoV of the stitched video frame was the FoV of the main view (Camera 1) augmented by FoVs of the four side views (Cameras 2–5). For each side-view camera, the FoV of each side-view was geometrically transformed to align it to Camera 1 through a homography The video stitching algorithm was implemented on a Linux laptop that had an Intel Xeon E3-150M v5 processor with 4 cores, and 32 GB of main memory. Multi-threading was leveraged to accelerate the computation. At the time of the experiment, this implementation of the video stitching algorithm achieved a processing rate of 25 stitched video frames per second.The bean drop task was designed to emulate the laparoscopic surgical task of moving from one location to another during surgery . In a usIn the modified bean drop task, a subject was asked to move eight separate beans, each to a separate location in the trainer box. In addition, two types of cups as shown in We laid out a set of nine inverted cups and two saucers in the pattern seen in Two surgical graspers are provided to perform the modified bean drop task so that the subject may use both hands to perform the task. During the transfer of a bean from a saucer to a cup, it may be dropped accidentally. The subject may pick up a dropped bean to complete the task with some timing penalty. However, if the subject cannot see the dropped bean via the visualization system or cannot reach it via the provided ports, that bean is deemed lost during transfer. When a bean is lost, the protocol requires the subject to move on to transfer the next bean to the required numbered cup. The completion time for each of the four tasks and the number of beans dropped during each task would be recorded for subsequent data analysis.A standard FLS laparoscopic trainer box designed for practicing laparoscopic procedures was used to validate and evaluate the effectiveness of the proposed micro-camera array. The trainer box is a low fidelity form of simulation but has known validity evidence for laparoscopic surgeries . By defaAs shown in We recruited 26 test subjects consisting of a medical student, surgical residents, and faculty of the University Hospital of the University of Wisconsin. The group consisted of 1 medical student, 3 first-year residents, 5 second-year residents, 6 third-year residents, 2 fourth-year residents, 3 fifth-year residents, and 6 faculty surgeons. The subjects were randomly recruited based on availability. The medical student had no prior experience with laparoscopic surgery. The first-year residents had less than a year of experience while the other residents had experience approximately equal to their years of residency. The faculty surgeons had extensive prior laparoscopic experience and were recruited based on the fact that they routinely perform minimally invasive (laparoscopic) surgery.Testing took place in the laparoscopic trainer room of the Simulation Center in the University Hospital over the course of 5 days. During testing, subjects were asked to stop by the testing facility at their leisure to perform a short laparoscopic training task to help test out a new visualization system. There was no restriction on how many subjects were allowed in the testing room, nor were subjects prohibited from watching other subjects perform the task, although they were discouraged from doing so.When a subject arrived at the testing facility, the subject was given a brief explanation of the difference between the two visualization systems and then given a description of the tasks to be completed. The subject was given a picture similar to After the explanation, the subject was directed to one of the two trainer boxes to begin the task. Each setup as seen in Each subject repeated the above procedure four times during the experiment using the two training boxes: twice with the traditional single-camera visualization system and twice with the proposed micro-camera array visualization system. The experiment protocol did not allow a subject to practice on the modified bean drop task in advance. To mitigate the concern that the subject will perform better in later trials (the third or the fourth) due to familiarity with the modified bean drop task in earlier trials, we chose to have the subject switch visualization systems after the first and third attempts. Specifically, the first and fourth attempts were performed with one visualization system and the second and third attempts were performed with the other. Furthermore, either type of visualization system was chosen alternatively for the first trial.The outcome of each modified bean drop task was the total completion time for completing the task. However, not all subjects completed all eight required sub-tasks. Sometimes a bean was dropped prematurely outside the destination cup and there was no way to recover it. When this happened, the protocol required the subject to abort the current sub-task and continue to the next one. As such, the completion time with aborted sub-tasks was shorter than if all eight sub-tasks had been completed successfully. Using the traditional laparoscope, there were a total of 26 aborted sub-tasks. Using the micro-camera array, the number of aborted subtasks increased to 45. Note that the total number of sub-tasks performed was To rectify the reduction of the completion time due to aborted bean drop, a penalty of 28 s was added to the measured completion time for each occurrence of a failed sub-task. The 28 s penalty was the averaged single sub-task completion time of all subjects for both treatments.In A two-sided paired T-test was applied to each of these distributions in Camera hopping is the act of removing and reinserting a laparoscope through the trocar to provide a better viewing angle during the procedure. In the modified bean-dropping test, we recorded multiple camera hopping operations when the laparoscope was used. There was no camera hopping when the micro-camera array was used. Manual inspection of the recorded videos found that on average subjects performed the camera hopping motion However, the time spent moving the camera can not be spent performing the task. In From Moreover, the completion time utilizing the micro-camera array consistently has a smaller standard deviation than that using the traditional laparoscope. This is also shown in The subjects recruited in this experiment are all familiar with using the laparoscope trainer box except the medical school student. All subjects are unfamiliar with using the micro-camera array viewing system. This familiarity and prior experiences with the traditional laparoscope may contribute to shorter completion time.There were a total of 71 aborted sub-tasks out of a total of 832 performed by all 26 subjects. The abort rate was approximately We observed that with the micro-camera array visualization system, some more experienced subjects used both hands to hold an additional surgical instrument to improve the overall efficiency of completing the modified bean drop task. This illustrates an important benefit of the micro-camera array visualization system. Namely, there will be no need to occupy a dedicated port for inserting the laparoscope and there will be no need to have an assistant to hold the laparoscope during the procedure.In the proof-of-concept design in this experiment, the cameras were chosen based on their small form factor to fit into the inside of a surgical trocar. The image quality of individual micro-cameras is inferior to that produced by a traditional single-camera laparoscope. Since the micro-camera array is hanging high, it provides a bird-eye view, rather than a close-up view as does the traditional laparoscope. In the next phase of this project, we plan to develop heterogeneous camera arrays that can simultaneously provide a global bird-eye view to facilitate navigation and close-up view to facilitate delicate surgical operations.In the interest of real-time video stitching, the video stitching algorithm does not compensate for the severe parallax distortion in the stitched video. The image of a laparoscopic surgical instrument may show discontinuity across the boundary between adjacent camera views when the surgical instrument is moving during the experiment. The default viewpoint is also fixed to the top-down view of the center camera. Hence, the subjects are unable to view the scene from an angle to acquire depth perception. In the following development, a multi-planar image stitching algorithm will be developed to mitigate the parallax anomaly. An any-view 3D dynamic rendering viewing system will also be implemented so that surgeons may shift their viewing angle from head-mounted augmented reality goggles to examine the surgical spots from an optimal angle.In this paper, an early performance evaluation of a prototype micro-camera array for laparoscopic surgery is reported. We conducted an experiment to validate a hypothesis that larger-FoV visualization can improve the efficiency of navigating laparoscopic surgical tools. In this experiment, human subjects including surgeons, residents, and a medical student were recruited to perform a modified bean drop task using laparoscopic trainer boxes equipped with the micro-camera array and a traditional laparoscopic camera, respectively. The time to complete each task was recorded and analyzed. We observed statistically significant performance improvement in terms of shorter time of completion. Future works will be on developing a laboratory prototype based on this proof-of-concept micro-camera array and corresponding computer vision algorithms to enhance its performance and usability.