A convolutional neural network-based multiobject detection and tracking algorithm can be applied to vehicle detection and traffic flow statistics, thus enabling smart transportation. Aiming at the problems of the high computational complexity of multiobject detection and tracking algorithms, a large number of model parameters, and difficulty in achieving high throughput with a low power consumption in edge devices, we design and implement a low-power, low-latency, high-precision, and configurable vehicle detector based on a field programmable gate array (FPGA) with YOLOv3 (You-Only-Look-Once-version3), YOLOv3-tiny CNNs , and the Deepsort algorithm. First, we use a dynamic threshold structured pruning method based on a scaling factor to significantly compress the detection model size on the premise that the accuracy does not decrease. Second, a dynamic 16-bit fixed-point quantization algorithm is used to quantify the network parameters to reduce the memory occupation of the network model. Furthermore, we generate a reidentification (RE-ID) dataset from the UA-DETRAC dataset and train the appearance feature extraction network on the Deepsort algorithm to improve the vehicles’ tracking performance. Finally, we implement hardware optimization techniques such as memory interlayer multiplexing, parameter rearrangement, ping-pong buffering, multichannel transfer, pipelining, Im2col+GEMM, and Winograd algorithms to improve resource utilization and computational efficiency. The experimental results demonstrate that the compressed YOLOv3 and YOLOv3-tiny network models decrease in size by 85.7% and 98.2%, respectively. The dual-module parallel acceleration meets the demand of the 6-way parallel video stream vehicle detection with the peak throughput at 168.72 fps. As urbanization accelerates, new infrastructure is empowering the development of smart cities while providing new impetus to the postepidemic economy. Satisfying the need for smooth traffic flow and improving traffic efficiency (reducing the number of vehicles on the road and increasing vehicle occupancy) are the goals of intelligent transportation systems. To achieve these goals, Bilal et al.  used HadMany detection algorithms have emerged for traffic information collection. Traditional detection algorithms such as optical flow  and backFPGAs are semicustom circuits that have a lower latency and higher parallelism capability than CPUs and have a lower power consumption and lower cost than GPUs. Compared to ASIC, they have a shorter design cycle, are more iterable, and are less costly. With the rapid development of deep learning frameworks, FPGAs are the best platform for the deep learning model forwarding inference acceleration .There are still many challenges in FPGA hardware acceleration. The performance of hardware acceleration is directly related to the on-chip resources of FPGAs. How to use limited hardware resources to design an efficient hardware acceleration architecture is a very important research problem. For the application of the YOLO series neural network, which is computationally intensive and has huge parameters, a high memory access frequency, and complex control logic, improving the acceleration performance involves two difficulties: optimizing the computing process and optimizing the memory exchange. In order to improve the accuracy and quantity of vehicle detection, networks with higher accuracy are needed. However, due to the huge amount of network parameters and computation, these networks will bring high resource and computation costs . How to We trained the YOLOv3 and YOLOv3-tiny networks using the UA-DETRAC dataset . Then, wA reidentification dataset was generated based on the UA-DETRAC dataset and used to train the appearance feature extraction network of the Deepsort algorithm with a modified input size to improve the vehicle tracking performance.We designed and implemented a vehicle detector based on an FPGA using high level synthesis (HLS) technology. At the hardware level, optimization techniques such as the Im2col+GEMM and Winograd algorithms, parameter rearrangement, and multichannel transmission are adopted to improve the computational throughput and balance the resource occupancy and power consumption. Compared with the other related work, vehicle detection performance with higher precision and higher throughput is realized with lower power consumption.Our design adopts a loosely coupled architecture, which can flexibly switch between the two detection models by changing the memory management module, optimizing the balance between the software flexibility and high computing efficiency of the dedicated chips.There have been many studies focusing on the FPGA-based convolutional neural network acceleration. However, the research of transplantation optimization for large models remains in theoretical research and fails to combine with specific scenarios for application deployment. Therefore, taking vehicle tracking and counting as the specific application scenario, we study the deployment of the YOLO series network acceleration on the FPGA side from the two aspects: the neural network compression and hardware accelerator design. In addition, we retrain the appearance feature extraction network of the Deepsort algorithm based on the self-generated dataset for vehicle tracking. The main contributions of this work are summarized as follows:The rest of this paper is organized as follows: YOLO was proposed by Joseph Redmon et al. in 2016. Compared with other networks in the YOLO series, YOLOv3 and YOLOv3-tiny achieves a good balance in accuracy and detection speed, and is easy to transplant to hardware.The network structure of YOLOv3 is similar to that of YOLOv3-tiny, except that it does not use max-pooling layers and uses the residual network, thereby constructing DarkNet-53 as the backbone with deeper network layers and one more prediction branch than YOLOv3-tiny, with a scale of j to track i is shown in (1), the spatial martingale distance is shown in (2), and the visual distance is shown in (3). The meanings of the parameters in the formula are shown in Deepsort  is an onNeural networks used in object detection, such as vehicle detection, usually require deeper network layers and more complex network structures. Therefore, such large-scale parameters and high computational complexity pose a challenge to the end-to-end deployment of DNNs.Many studies have proposed various methods for neural network compression and simplification, including network pruning, weight quantization, approximated computing and low-rank decomposition of the weight matrix. Han et al.  first prThere has been much research focusing on the FPGA-based acceleration of convolutional neural networks. Ma et al.  improvedThe above studies have made great contributions to deploying AI directly on edge devices, but as the models become more complex, research on the optimization of large models remains in theoretical studies and fails to be deployed in conjunction with specific scenario applications.The neural network acceleration contains complex operators and memory management modules, so using HDL (Hardware Description Language) to directly describe the framework has a long development cycle, making it difficult to explore the design space. HLS uses C/C++ to describe the framework from a high level. It greatly improves development efficiency due to the rapid conversion of high-level code to FPGA implementation . Many stCombined with the background knowledge in Model pruning is still the most effective method of neural network compression thus far, and it reduces the storage space and inference time required for the model by eliminating parameters from the model that have little impact on the detection effect .The workflow of the structured pruning based on the dynamic threshold of the binary search adopted in this paper is shown in W represents the trainable weight. The first term represents CNN training losses, Sparse regularization training first introduces a scaling factor for each channel, which is used to multiply with the output of that channel. The scaling factors are trained jointly with the network weights and are sparsely regularized during the training to identify insignificant channels. The objective function of the sparse regularization training is shown in (4), where The structure of the network after sparse regularization training is shown in x is a floating-point number, and Q is used to specify the base point position of the fixed-point number.When the model is trained in the GPU, it usually uses a 32-bit floating point to express the weight, gradient, and activation value of the network. Using floating-point operations increases the overhead of the computational unit; thus, currently, lower bit-width fixed-point numbers are usually used for the inference process of neural networks. We carried out the dynamic 16-bit fixed-point quantization for weight bias parameters, feature mapping, and intermediate results in three quantization stages . The conw and the exponent character Q, as shown below:For a given fixed-point number, its actual floating-point value is a function of the bit width Q values are analyzed for each layer dynamically using the approach shown in Equations (8) and (9), so that the absolute error sum of the original value of the weight bias and the quantized value is minimized. l-th layer weights and biases, respectively, and l-th layer weights and biases, respectively.In the stage of quantifying the weight values and bias values, the optimal Q value for each layer of the input–output feature map, and the optimal Q value is calculated as shown in Equations (10) and (11). For example, the RGB value of the input image is scaled to the  interval in the preprocessing stage, and Q = 14 can be used to quantize the input of the first layer when the bit width w = 16.In the stage of quantization of inputs and outputs between layers, we find the optimal Q value for each layer of intermediate data by using the approach shown in (12).In the stage of quantifying the intermediate results, we find the best The Deepsort algorithm uses a small CNN to extract the appearance features of the detected target. For each detected frame, the similarity of the detected target in the current frame is compared with the appearance features previously saved. The original network is used to extract the appearance features of pedestrians; thus, the default width × height of the input is To adapt the Deepsort algorithm to vehicle tracking, we need a large dataset to train the vehicle appearance feature extraction network. Currently, there are few reidentification datasets based on the roadside traffic monitoring of vehicles. Therefore, based on the vehicle recognition information and detection frame information provided by UA-DETRAC, we generated the vehicle reidentification dataset, REID-UA-DETRAC. To ensure the effectiveness of training, we selected vehicles that appeared more than 200 times as valid data. The generated dataset contained 1053 identities and 606,007 images in total. Part of the generated dataset is shown in Based on the YOLOv3 and YOLOv3-tiny models, we designed the architecture of the vehicle detection accelerator. All kinds of calculations required by the model are optimized and encapsulated into the corresponding calculation modules in the FPGA. The overall architecture of the accelerator is shown in The accelerator consists of a host computer and an FPGA. The main tasks of the host are image preprocessing, data quantization, nonmaximal suppression, and Deepsort task scheduling. The host uses the controller to schedule the flow of data and uses the memory manager to manage the interaction between DRAM and DMA. The FPGA is responsible for the accelerated calculation of various computation-intensive tasks. The host loads the configuration information of the current model at the beginning, and stores the pre-quantized weight and bias data of the model in a continuous memory. Then, the host extracts the input video into frame images and sends them to the controller module in sequence. First, the control module sends the image to the data quantification module. Then, it transfers the quantized image, the weight, and bias data of the current layer to the FPGA on-chip memory through the optimized transmission method of ping-pong double buffering and multi-channel transmission. After the acceleration of a specific computing module, the result is sent back to the off-chip DRAM through the above method. After completing the prediction of an image, the host performs the NMS (Non Maximum Suppression) operation and transmits the result to the Deepsort tracking module. Finally, it draws the tracking result into a new video stream in real time.DNNs contain a large number of parameters; however, the on-chip Block Random Access Memory (BRAM) cannot carry all the data. Storing all the parameters in the off-chip Dynamic Random Access Memory (DRAM) will significantly increase the access latency. Therefore, we use DRAM to store image data and weight data. During the operation, the data blocks involved in the operation are transmitted to BRAM, and the results are written back to DRAM after the operation is completed.The accelerator we proposed encapsulates the operators required by the YOLOv3 and YOLOv3-tiny models. In the stage of designing the overall architecture, we take into account the configurability of the accelerator, and design two memory management drivers for the two models, respectively. Such a design can switch detection models by adaptively or manually switching memory management drivers based on real-time traffic and weather conditions. When the detection environment is poor, using YOLOv3 can achieve a higher accuracy. However, when the traffic increases, the calculation using YOLOv3 takes a long time, and the real-time detection effect can be achieved by using YOLOv3-tiny. Based on this, when the traffic flow is high (the detection results of 10 consecutive frames are more than 15 vehicles), the software automatically switches to use the memory management driver of YOLOv3-tiny to improve the detection speed. When the detection environment is not ideal , we can manually switch to the YOLOv3 model to obtain a higher detection accuracy. We read the parameters of both models into the specified memory in the program loading stage, and the switching action is after the end of a picture detection, so there is almost no switching delay. According to the accuracy priority and speed priority, the accelerator can better adapt to the change in the detection environment. The following is an introduction to the two memory driver designs.Since the input of YOLOv3 and YOLOv3-tiny is the output of the previous layer except for the first layer, we use the strategy of memory interlayer multiplexing to reduce DRAM consumption.The interlayer multiplexing strategy adopted by YOLOv3 is the same as that of YOLOv3-tiny. Since the YOLOv3 network has more layers and a more complex model, its memory management module consists of five caches of size The convolution operation requires the parallel calculation of the results of all channels and then summing them. The original weight parameters are stored in memory in row priority order, which requires more access times. According to the block division of the weight of each layer, we reordered the weight parameters of each layer in advance, which reduced the accessing delay time during the operation . The parTaking the weight parameters of the 12th layer of YOLOv3-tiny as an example, as shown in We adopt the multichannel transmission strategy shown in The input characteristic diagram is transmitted by n channels in parallel, and the output characteristic diagram is transmitted by m channels in parallel. After the current layer is partitioned, each incoming channel reads the pixel blocks corresponding to After an exhaustive architecture search, we give the values of these parameters to determine the final accelerator architecture: For the convolutional layer, The The architecture determined by the above parameter reduces the number of feature maps transmitted by each channel from Due to the dependence of data transfer between two adjacent layers and the convolutional module needing to traverse the output of all channels of the previous layer, the interlayer flow optimization cannot be carried out. However, there is no data dependence among all channels in the calculation of the pooling layer, so some data from the upper layer can be used for direct calculation. Therefore, we fuse the convolution module with the pooling module to reduce the triple access of the two operations to one, as shown in For the convolution module designed with the Im2col+GEMM algorithm, we use the in-layer pipeline design shown in The convolution operation is used for feature extraction of images, and its computational complexity is ribed by . After oCNN computing requires a large amount of memory, but the FPGA’s on-chip storage resources cannot meet the requirement of storing such a large amount of data at a time . TherefoWe deeply analyze the characteristics of convolution operations with kernel sizes of col+GEMM  and Winocol+GEMM  algorithIm2col+GEMM: The Im2col+GEMM algorithm reduces the time complexity of the convolution operation from The GEMM algorithm needs to stretch the convolution kernel and feature map into the matrix form. This transformation process not only includes the delay increase caused by the multiple access but also requires more storage space to temporarily store the matrix form of the feature map for which the kernel size is larger than The space complexity is proportional to the convolution kernel size Winograd convolution:For the convolution operation with a convolution kernel size of Y to represent the output of this function, which can be expressed in the form of Equation Y=AW represents the convolution filter, I represents the input feature map, G is the convolution kernel transformation matrix of size A is the output feature map transformation matrix of size B is the input feature map transformation matrix of size In Equation , W repreMax-pooling is a downsampling operation that selects the largest value in an image region as the pooled value in that region based on the stride and the pooling size . Poolingn pixels at the same position of the feature map are read in parallel in each clock cycle, and the pooling result is obtained after the n paths and written into the output cache.The max-pooling module is implemented by the comparator. N pooling modules run in parallel, and the result of N feature maps can be obtained after the Each max-pooling module directly reads the convolution result of the previous layer from on-chip memory. They compare the read result with the predefined minimum value MIN and store the larger value in register Reg, while the counter Upsampling is an image magnification technique that improves the image resolution by interpolation. This operation first takes every pixel in the image and makes four copies of it, doubling the width and height of the output feature image.Because the upsampling calculation process is simple and the calculation scale is small, we use the combinatorial logic circuit to implement the module. The optimization idea we use is similar to the max-pooling module, so we will not go into detail here.Batch normalization is often used in DNNs to accelerate network convergence after linear calculation and before nonlinear activation . To reduLet Equation  can be sSubstituting (17) into (23), we obtain (24).Equation  is rearrBy integrating BN with a convolution operation, computing resources and delay can be reduced.We designed and simulated the proposed accelerator to verify the effectiveness of the proposed optimization method. The training and pruning quantization were completed by the NVIDIA Tesla V100 platform. The detection inference was implemented by the CPU+FPGA heterogeneous platform. The chip we used is ZYNQ XC7Z035-FFG676-2. We designed the IP cores of YOLOv3 and YOLOv3-tiny accelerators using Xilinx Vivado HLS 2021.2, and used Vivado 2021.2 for the synthesis and layout.We used the UA-DETTAC dataset to train the YOLOv3 and YOLOv3-tiny detection models. It is a large-scale dataset used for vehicle detection and tracking, covering 1.21 million detection frames of 8250 vehicles under different weather and road conditions.During training, the four categories of the dataset were merged into a single category, called car. The same training parameters were selected in the training stages of the two models. After basic training and pruning for the YOLOv3 and YOLOv3-tiny models, the specific performance indices are shown in According to the data in To show the detection effect more intuitively, we selected an image from the test set to be detected using the models before and after compression, as shown in The number of vehicles detected by each model is shown in The UA-DETRAC dataset is derived from the road surveillance video in real scenarios. Therefore, the images in the test set can be converted to the video with a frame rate of 24 fps for the tracking test of the Deepsort algorithm. A comparison of the tracking results before and after training using the reidentification dataset is shown in We further used the MOT-Metrics tool , combineThe MVI_40701 video stream was captured in the daytime peak traffic flow scene and shot from a forward overlooking angle. All evaluation indices show that the RE-ID Deepsort algorithm has better performance than the Deepsort algorithm in tracking vehicles.The MVI_40771 video stream was captured in the peak traffic flow scene at night and shot from a forward overlooking angle. Compared with the Deepsort algorithm, the RE-ID Deepsort algorithm improved significantly in several evaluation indices, especially in reducing the number of IDs. The experimental results indicate that the model proposed in this paper has an obvious improvement effect on vehicle detection in night scenes.The MVI_40771 video stream was shot on a rainy day with heavy traffic. The shooting angle was a side view. A large number of small cars were covered by large cars in this video stream; thus, all tracking indices are inferior to those of the previous two video streams. Except for the same MOTP, RE-ID Deepsort is better than Deepsort in other indicators.We use the YOLOv3 and YOLOv3-tiny models after pruning and combine them with the RE-ID Deepsort algorithm to conduct vehicle tracking counting experiments. The scenarios we tested included peak traffic under the day, night, and rainy conditions. In these scenarios, vehicles move slowly, and the vehicles are relatively close, which easily generates occlusion. A car located between two large vehicles will cause missed detection. Especially in the third scenario, the camera is located on the side of the road; thus, cars in the middle of the road are almost completely covered by large cars on the side road when moving slowly, resulting in poor detection results. The accuracy rates of YOLOv3 and YOLOv3-tiny were 96.15% and 92.3% in daytime conditions, 94.0% and 92.0% in nighttime conditions, and 81.8% and 75.8% in rainy conditions, respectively.To cope with the demand of detecting multiple video streams at one intersection in practical applications, we implemented a design of deploying multiple accelerator modules on-chip xc-7z035-ffg676-2. The experimental results shown in Our acceleration architecture currently supports the acceleration of three operators: convolution, upsampling, and max-pooling. Many auxiliary functions come from darknet, so it has good support for YOLO series networks. It also supports YOLOv2 and YOLOv2-tiny. Since their detection performance is not as good as YOLOv3 and YOLOv3-tiny, we do not compress them. The inference speeds of the original YOLOv2 and YOLOv2-tiny are 433 ms and 72 ms, respectively. It can support YOLOv4 network by adapting the memory management driver. In order to adapt to YOLOv5, the focus structure needs to be optimized.By adapting memory management drivers and auxiliary functions such as load_network and get_detections, our acceleration architecture can be applied to other networks such as SSD, ResNet, and MobileNet.We implemented a scalable vehicle detector based on the FPGA and the YOLO object detection algorithm using HLS. In terms of model compression, we incorporate the dynamic threshold structured pruning strategy based on the binary search and the dynamic INT16 fixed-point quantization algorithm to significantly reduce the model size and computation. In terms of vehicle tracking, we generated a reidentification dataset based on the UA-DETRAC dataset and used it to train the appearance feature extraction network of the Deepsort algorithm with a modified input size to improve the vehicle tracking performance. For the two convolution operations, we designed fast convolution engines based on the Winograd and GEMM algorithms, respectively. For max-pooling and upsampling computation in YOLOv3 and YOLOv3-tiny models, we use combinatorial logic to implement parallel computation to improve the computational efficiency.Compression acceleration of neural networks still has many worthy research directions. In the future, the search for network structures suitable for solving specific problems using the optimization scheme of neural architecture searches (NASs) can be used as a research direction for network compression. In terms of hardware acceleration, the current neural network accelerators usually only support a few network frames, and the generalization of acceleration frames is also worthy of further research.


Training of neural networks can be reformulated in spectral space, by allowing eigenvalues and eigenvectors of the network to act as target of the optimization instead of the individual weights. Working in this setting, we show that the eigenvalues can be used to rank the nodes’ importance within the ensemble. Indeed, we will prove that sorting the nodes based on their associated eigenvalues, enables effective pre- and post-processing pruning strategies to yield massively compacted networks (in terms of the number of composing neurons) with virtually unchanged performance. The proposed methods are tested for different architectures, with just a single or multiple hidden layers, and against distinct classification tasks of general interest. In standard neural network training one seeks to optimise the weights that link pairs of neurons belonging to adjacent layers of the selected architecture4. This is achieved by computing the gradient of the loss with respect to the sought weights, a procedure which amounts to operate in the so called direct space of the network5. Alternatively, the learning can be carried out in reciprocal space: the spectral attributes  of the transfer operators that underlie information handling across layers define the actual target of the optimisation. This procedure, first introduced in5 and further refined in6, enables a substantial compression of the space of trainable parameters. The spectral method leverages on a limited subset of key parameters which impact on the whole set of weights in direct space. Particularly relevant, in this respect, is the setting where the eigenmodes of the inter-layer transfer operators align along random directions. In this case, the associated eigenvalues constitute the sole trainable parameters. When employed for classifications tasks, the accuracy displayed by the spectral scheme restricted to operate with eigenvalues is slightly worse than that reported when the learning is carried in direct space, for an identical architecture and by employing the full set of trainable parameters. To bridge the gap between conventional and spectral methods in terms of measured performances, one can also train the elements that populate the non trivial block of the eigenvectors matrix5. By resorting to apt decomposition schemes, it is still possible to contain the total number of trainable parameters, while reaching stunning performances in terms of classification outcomes6.Automated learning via deep neural networks is gaining increasing popularity, as a ductile procedure to address a widespread plethora of interdisciplinary applications7) with recorded performance comparable to those displayed by their unaltered homologues, after a proper round of training7. More specifically, after a first round of training which solely acts on the eigenvalues, one can identify the most relevant nodes, as follows the magnitude of the associated eigenvalues. Since the first training stage is just targeted to eigenvalues, the eigenvectors obtained after pruning are still bearing reflexes of the random initialization and thus represent a sort of “winning ticket”7. In this respect, according to the above reasoning, the proposed two stages strategy can be seen as a novel and efficient way to discover optimal sub-networks.In this paper we will discuss a relevant byproduct of the spectral learning scheme. More specifically, we will argue that the eigenvalues do provide a reliable ranking of the nodes, in terms of their associated contribution to the overall performance of the trained network. Working along these lines, we will empirically prove that the absolute value of the eigenvalues is an excellent marker of the node’s significance in carrying out the assigned discrimination task. This observation can be effectively exploited, downstream of training, to filter the nodes in terms of their relative importance and prune the unessential units so as to yield a more compact model, with almost identical classification abilities. The effectiveness of the proposed method has been tested for different feed-forward architectures, with just a single or multiple hidden layers, by invoking several activation functions, and against distinct datasets for image recognition, with various levels of inherent complexity. Building on these findings, we will also propose a two stages training protocol to generate minimal networks  which outperform those obtained by hacking off dispensable units from a large, fully trained, apparatus. This strategy can be seen as an effective way to discover sub-networks  in direct space, as a function of the underlying spectral quantities. In formulae, one gets:The loss function can thus be minimized by acting on the spectral parameters, i.e. the ensemble made of non trivial eigenvalues and/or the associated eigendirections. A straightforward calculation, carried out in the annexed  j. Then, the activity i reads:To help comprehension denote by 6, this can be rationalized as the artificial analogue of the homeostatic plasticity, the strategy used by living neurons to maintain the synaptic basis for learning, respiration, and locomotion9.The eigenvalues Starting from this background, we shall hereafter operate within a simplified setting which is obtained by imposing 11.Generally speaking, it is possible to ideally group various approaches for network compression into five different categories: Weights Sharing, Network Pruning, Knowledge Distillation, Matrix Decomposition and Quantization12. A refined approach is proposed in Bat et al.13 where a virtual infinitely deep neural network is considered. Further, in Zhang et al.14 an Weights Sharing defines one of the simplest strategies to reduce the number of parameters, while allowing for a robust feature detection. The key idea is to have a shared set of model parameters between layers, a choice which reflects back in an effective model compression. An immediate example of this methodology are the convolutional neural networks15 proposed an iterative pruning algorithm that exploits a continuously differentiable version of the 16 focused on pruning convolutional filters, so as to achieve better inference performances (with a modest impact on the recorded accuracy) in a transfer leaning scenario. Starting from a network fine-tuned on the target task, they proposed an iterative algorithm made up of three main parts: (1) assessing the importance of each convolutional filter on the final performance via a Taylor expansion, (2) removing the less informative filters and (3) re-training the remaining filters, on the target task. Inspired by the pioneering work in7, Pau de Jorge et al.17 proved that pruning at initialization leads to a significant performance degradation, after a certain pruning threshold. In order to overcome this limitation they proposed two different methods that enable an initially trimmed weight to be reconsidered during the subsequent training stages.Network Pruning is arguably one of the most common technique to compress Neural Network: in a nutshell it aims at removing a set of weights according to a certain criterion . Chang et al.18. In its simplest version Knowledge Distillation is implemented by combining two objective functions. The first accounts for the discrepancy between the predicted and true labels. The second is the cross-entropy between the output produced by the examined network and that obtained by running a  trained model. In19 Polino et al. proposed two approaches to mix distillation and quantization (see below): the first method uses the distillation during the training of the so called student network under a fixed quantization scheme while the second exploits a network (termed the teacher network) to directly optimize the quantization. Mirzadeh et al.20 analyzed the regime in which knowledge distillation can be properly leveraged. They discovered that the representation power gap of the two networks (teacher and student) should be bounded for the method to yield beneficial effects. To resolve this problem, they inserted an intermediate network (the assistant), which sits in between the teacher and the student, when their associated gap is too large.Knowledge Distillation is yet another technique, firstly proposed by Hinton et al.21 proposed a matrix decomposition method for transfer learning scenario. They showed that decomposing a matrix taking into account the activation outperforms the approaches that solely rely on the weights. In22, Novikov et al. proposed to replace the dense layer with its Tensor-Train representation23. Yu et al.24 introduced a unified framework, integrating the low-rank and sparse decomposition of weight matrices with the feature map reconstructions.Matrix Decomposition is a technique that remove redundancies in the parameters by the means of a tensor/matrix decomposition. Masana et al.25 defined an algorithm that quantize the model by minimizing the reconstruction error for inputs sampled from the training set distribution. The same authors also claimed that their proposed method is particularly suited for compressing residual network architectures and that the compressed model proves very efficient when run on CPU. In Banner et al.26 a practical 4-bit post-training quantization approach was introduced and tested. Moreover, a method to reduce network complexity based on node-pruning was presented by He et al. in27. Once the network has been trained, nodes are classified by means of a node importance function and then removed or retained depending on their score. The authors proposed three different node ranking functions: entropy, output-weights norm (onorm) and input-weights norm (inorm). In particular, the input-weights norm function is defined as the sum of the absolute values of the incoming connections weights. As we will see this latter defines the benchmark model that we shall employ to challenge the performance of the trimming strategy here proposed. Finally, it is worth mentioning the Conditional Computation methods30: the aim is to dynamically skip part of the network according to the provided input so as to reduce the computational burden.Quantization, as also mentioned above, aims at lowering the number of bits used to represent any given parameter of the network. Stock et al.6 and exploits the fact that eigenvalues are credible parameters to gauge the importance of a given node among those composing the destination layer. In short, our aim is to make the network more compact by removing nodes classified as unimportant, according to a suitable spectral rating.Summing up, pruning techniques exist which primarily pursue the goal of enforcing a sparsification by cutting links from the trained neural network and have been reviewed above. In contrast with them, the idea of our method is to a posteriori identify the nodes of the trained network which prove unessential for a proper functioning of the device and cut them out from ensemble made of active units. This yields a more compact neural network, in terms of composing neurons, with unaltered classification performance. The method relies on the spectral learningIn order to assess the effectiveness of the eigenvalues as a marker of the node’s importance  we will consider a fully connected feed-forward architecture. Applications of the explored methods will be reported for 31, composed by greyscale images of dimension 32  which are still depicted with a greyscale with dimension 33 a richer dataset composed by For our test, we used three different datasets of images. The first is the renowned MNIST database of handwritten digits27. Following this assessment, nodes are progressively removed from the trained network, depending on the imposed percentile, and the ability of the trimmed network to perform the sought classification (with no further training) tested. We choose this particular type of trimming as a benchmark to our spectral pruning technique for the following reasons. First, it also amount to removing nodes from the collection, and not just sparsify the weight of the associated transfer matrices. Then, both approaches build on the concept of nodes ranking, as obtained from a suitable metric, which is respectively bound to direct vs. spectral domains. The abovementioned procedure is repeated 5 times and the mean value of the accuracy plotted in the orange curve of Fig. In Fig. The blue curve Fig. 7. As a matter of fact, the initial training of the eigenvalues uncovers a sub-network that, once trained, obtains performances comparable to the original model. More specifically, the uncovered network can be seen as a winning ticket7. That is, a sub-network with an initialization particularly suitable for carrying out a successful training.At variance, the green curve in Fig. Next, we shall generalize the analysis to the a multi-layer setting  in the description) is acknowledged as a post processing method, in that it acts on a multi-layered network downstream of training. The other (referred as (i)) is based on a sequence of two nested operations. First the eigenvalues are solely trained. After the spectral pruning took place, a second step in the optimization path seeks to adjust the entries of the eigenvectors that populate a trimmed space of reduced dimensionality. The total number of trained parameters is small as compared to that involved when the pruning acts as a post processing filter. Despite that, the two steps pre-processing protocol yields compact devices which outperform those obtained with a single post-processing removal of the unessential nodes.27. In principle, one could consider to diagonalizing the transfer operators as obtained after a standard approach to the training and make use of the computed eigenvalues to a posteriori sort the nodes relevance. This is however not possible as the transfer operator that links a generic layer k to its adjacent counterpart As a benchmark model, and for a neural network trained in direct space, we decided to rank the nodes importance based on the absolute value of the incoming connectivity. This latter appeared as the obvious choice, when aiming at gauging the local information flow in the space of the nodes, see alsoSumming up, by reformulating the training of neural networks in spectral space, we identified a set of sensible scalars, the eigenvalues of suitable operators, that unequivocally correlate with the influence of the nodes within the collection. This observation translates in straightforward procedures to generate efficient networks that exploit a reduced number of computing units. Tests performed on different settings corroborate this conclusions. As an interesting extension, we will show in the We detail here the spectral procedure to make a trained network smaller, while preserving its ability to perform classification.k to layer j is modulated by a multiplicative scaling factor i, j) pair. Then, all incoming  activities reaching the destination node i are summed together and further weighted via the scalar quantity i of layer i on layer To introduce the main idea of the proposed method, we make reference to formula  and assuAs we will clarify, the latter setting translates in a post-training spectral pruning strategy, whereas the former materializes in a rather efficient pre-training procedure. The non linear activation function as employed in the training scheme leaves a non trivial imprint, which has to be critically assessed.N neurons organized in M can be very small) which performs equally well than its fully trained analogue made of N computing nodes.(i) As a first step, we will begin by considering a deep neural network made of N neurons organized in (ii) We begin by constructing a deep neural network made of To evaluate the performance of the proposed spectral pruning strategies  neural network assessed on the test-set. The codes employed, as well as a notebook to reproduce our results, can be found in the public repository of this project (see Data availability section).In all explored cases, the pruning is realized by imposing a threshold on the reference indicator . Pointedly, the respective indicator is extracted for every node in the arrival layer. Then a percentile Supplementary Information.


With the development of artificial intelligence, facial expression recognition has become an important part of the current research due to its wide application potential. However, the qualities of the face features will directly affect the accuracy of the model. Based on the KDEF face public dataset, the author conducts a comprehensive analysis of the effect of linear discriminant analysis (LDA) dimensionality reduction on facial expression recognition. First, the features of face images are extracted respectively by manual method and deep learning method, which constitute 35-dimensional artificial features, 128-dimensional deep features, and the hybrid features. Second, LDA is used to reduce the dimensionality of the three feature sets. Then, machine learning models, such as Naive Bayes and decision tree, are used to analyze the results of facial expression recognition before and after LDA feature dimensionality reduction. Finally, the effects of several classical feature reduction methods on the effectiveness of facial expression recognition are evaluated. The results show that after the LDA feature dimensionality reduction being used, the facial expression recognition based on these three feature sets is improved to a certain extent, which indicates the good effect of LDA in reducing feature redundancy. With the development of the information society, as one of the subjects for human information exchange, the accurate recognition of facial expression will help to improve the efficiency of people's information exchange and to save time and cost. As one of the most primitive ways of information interaction, facial expression has been preserved to this day and plays an irreplaceable role. People's facial expressions can reveal their attitudes to the received information, the emotions toward people or things, and even their inner thoughts. Therefore, the technology of facial expression recognition can be applied to various scenarios, including human-computer interaction, as well as the security and medical fields, such as public security control, fraud behavior analysis, and patient emotion analysis. The method of facial expression recognition has become a research hotspot in the field of computer vision.Facial expression recognition can be roughly described as analyzing the data of face images by computer, extracting the useful feature information for recognition, and using this information to determine the category of the images. However, as the facial features are extracted and constructed by a variety of feature extraction methods, they are often of high dimensions. If the high-dimensional features are directly used in the subsequent recognition tasks, then it will bring great difficulties for the recognition task. The reason is that although the high-dimensional features contain the complete face information, they have a large number of redundant features. As the number of samples increases, the redundant information will greatly increase the computational resource overhead but reduce the performance of the recognition model. When the amount of data cannot support the feature dimension, a “dimension disaster” will occur. The performance of the recognition model drops greatly or even becomes ineffective. Therefore, it is a key step to reduce the dimension of high-dimensional features in face recognition technology, which has extremely important practical significance.The accurate recognition of facial expression recognition is inseparable from the extraction of multidimensional features, so redundant features are inevitable. Although the LDA feature dimensionality reduction method has been widely used in the field of face recognition, the previous work had not analyzed the dimensionality reduction effect of different types of feature sets on expressions. Based on the KDEF face public dataset, this study constructs three types of feature sets such as face traditional artificial feature sets, deep learning feature sets, and hybrid feature sets. Furthermore, it performs LDA feature dimension reduction respectively and uses machine learning models to classify the facial expressions. The experimental results show that dimensionality reduction of hybrid features can achieve the best accuracy of facial expression classification, which shows the importance of multidimensional hybrid feature extraction for machine learning tasks.In this study, it introduces the positioning of human facial key points, the definition of action units, and the construction of the coding system in The key of facial expression recognition is to construct the concise and accurate facial feature descriptors. The method based on the complete face recognition generally uses the traditional feature descriptors, such as Gabor , local bThe features of these facial AUs and the main regions they control can not only be used to distinguish faces but also to reflect the related facial expressions . Since tIn the process of facial expression recognition, in order to extract more comprehensive face features, the feature dimension is often too high because of the high complexity of facial image data. Therefore, it is usually necessary to reduce the dimension of the extracted features first to reduce the feature redundancy. Principal component analysis (PCA)  and lineAlthough the LDA feature dimensionality reduction method has been widely used in the field of face recognition, the impact of LDA feature dimensionality reduction on the face recognition under different feature sets has not been evaluated. In this regard, according to the KDEF face public data set, this study conducts a comprehensive analysis for the effect of facial expression recognition based on LDA feature dimensionality reduction, including traditional artificial features, deep learning features, and the hybrid features. First, multidimensional feature extraction is performed. For traditional artificial features, we extract face AU information according to the facial action coding system and construct a 35-dimensional AU feature set. For deep learning features, we use the facial feature extractor based on FaceNet pretraining to extract the deep facial features and construct a 128-dimensional feature set. Second, LDA dimension reduction is performed on the extracted features respectively. Then, machine learning models, such as Naive Bayes and decision trees, are used to analyze the effects of facial expression recognition before and after LDA feature dimensionality reduction. Finally, we compare the effects of PCA and LDA feature dimensionality reduction methods on facial expression recognition. The specific workflow is shown in In order to fully express the face features, we use different feature extraction methods to extract two aspects of face features according to the KDEF dataset: one is the AU feature based on the artificial methods and the other is the deep feature based on the deep learning.We adopt the method of AU feature extraction in OpenFace  to extraOpenFace describes the recognizable AUs in two scoring methods: (1) existence: if the AU is visible on the face, then 0 is absent and 1 is present; (2) intensity: if the AU has the intensity score on the interval , then 0 is absent and 5 is present at the maximum intensity. OpenFace is able to judge the existence of 18 recognizable AU subsets and score the intensity of 17 AUs except for AU28. Therefore, each face image can form 35-dimensional AU features, as shown in There are generally two ways to select the initial face key points. The first one is to take the mean value of real key points of all images as the initial face key points. The second one is to randomly select the real key points of other images as the initial key points of the current image. This study chooses the latter method to select the initial face key points.When a tree is constructed and multiple images are input into the current tree, each image will fall into a leaf node. Calculate the differences between the current face key points and the real face key points of each image, and take the mean value of the differences of all images in the same leaf node as the residual saved by the leaf node.Update the current face key points of each image as the sum of the current face key points and the residual.Repeat steps 2 and 3 until enough trees are established. In that way, the last current face key points can be used as the real face key points.In order to enable the deep learning model to focus on the face and better characterize the face, we use the method described in literature  to extraAccording to the above algorithm, we extract 68 face key points for each face image and use these key points to perform the affine transformation correction on the face, so that the corners of the eyes and the nose are close to the horizontal position. It is shown in After correcting the face image, we use the face feature extractor in OpenFace to extract deep face features. This feature extractor is based on the pretraining deep learning model FaceNet , which iOpenFaces uses 500,000 images to train FaceNet, including the two largest labeled face recognition datasets, CASIA-WebFaces , and Facc, in which any sample xi is an n-dimensional vector, and yi is its category, yi ∈ {n1, n2, ..., nk} , with a total of k categories. Nj is defined as the number of Class j sample, Xj is defined as the set of Class j sample, μj is defined as the mean vector of Class j sample, and ∑j is defined as the generalized covariance matrix of Class j sample.Due to the diversity of facial expressions, recognizing facial expressions is a multiclassification task, so it needs to use the multiclassification promotion form of LDA. For the problem of multiclassification, we assume the dataset W=, then the optimization objective is as follows:Sw is the intraclass scatter matrix:Assuming that the matrix formed by the projected hyper-plane basis is Sb is the interclass scatter matrix:In equation ,(4)μ=1m∑J(W) is a matrix, the diagonal element product is used to replace the matrix for optimization:Since J(W) will obtain the maximum value. As the maximum value of the Rayleigh quotient wTSbw/wTSww is the maximum eigenvalue of Sw−1Sb, the matrix W formed by the eigenvectors corresponding to the product of the largest d eigenvalues is the mapping result after feature dimensionality reduction.In equation , the rigS= is an item to be classified and each si is a feature attribute of S. According to the existing category set C=, we can calculate the probability of all categories under the characteristic condition of S and select the category with the highest probability as the category label of S. In this way, we can obtain the equation definition of Naive Bayes classifier as shown in the following equation:In order to evaluate the influence of the feature set on the face recognition after LDA dimensionality reduction, we rely on the several machine learning methods for facial expression recognition, including Naive Bayes classification model (NB), the decision tree classification model (DT), K-nearest neighbor (KNN), random forest (RF), and support vector machine (SVM). Among them, the Naive Bayes model  outputs V(S) is the category label of S, p(S) is a constant for all categories, and p(ci) is the category prior probability. p(s1|ci), p(s2|ci), ..., p(sn|ci) is the conditional probability of each feature attribute in S under the condition of category ci. All of these can be obtained from the training sample sets.In equation , V(S) isS is the set of training samples, |S| is the number of training samples, and A is any feature of the sample. When the samples are divided into n different classes C1, C2,…, Cn, and the number of samples in these classes is marked as |C1|, |C2|,…, |Cn|, then the probability that any sample S belongs to class Ci is as follows:The decision tree classification model  consistsThen, the total information entropy is as follows:A is as follows:The information entropy of samples divided in accordance with Feature A on the set S is as follows:The information gain of Feature A is the sum of all possible values v of attribute A, Sv is the subset of sample features A with v value in S, and |Sv| is the number of samples in Sv. According to the theory of information theory, the ID3 algorithm uses “information gain” to measure uncertainty; that is, the greater the information gain, the smaller the uncertainty and the better the division effect.In equation , ∑A is tThe experimental conditions in this study are CPU: Intel(R) Core (TM) i5-10210U, RAM: 16.0 GB, and the programming language is Python 3.6.The experiments in this study are based on the KDEF public face dataset . The KDEN denotes the number of categories, TP is true positive, FP is false positive, and FN is false negative.We randomly select 10% of the identifiable samples as the test set and the rest as the training set. The recognition effect is measured by the multiclassification evaluation metrics, including microprecision (micro-P), microrecall (micro-R), micro-F1, macroprecision (macro-P), macrorecall (macro-R), and macro-F1, which is shown in equations –16), wh, wh16), In order to test the effect of LDA feature dimensionality reduction on the accuracy of facial expression recognition, we respectively extract the AU feature and the 128-dimensional feature from the images in the dataset as two different feature sets and regard the hybrid feature set as the third feature set. For the three feature sets, LDA was used to reduce the dimensions to 7 dimensions, and the training set of the machine learning method is used to test its classification effect. Therefore, the experiment can be divided into three parts: analysis of LDA dimensionality reduction effect based on AU features, analysis of LDA dimensionality reduction effect based on 128-dimensional face deep features, and analysis of LDA dimensionality reduction effect based on hybrid features.The facial AU feature is obtained by dividing the face into regions and counting the influence of each expression on different regions. It contains rich expression information, but it may also have redundancy that affects the recognition results. Therefore, we adopt several machine learning methods for facial expression recognition on basis of the AU feature and compare the recognition effects before and after LDA feature dimensionality reduction. The results are listed in Macro-F1 is taken as an example in The feature extraction based on the deep learning model can perform a higher-level abstract expression. In contrast to the local AU features, the semantic information of these features is richer, but the detailed location information is relatively brief. The facial expression recognition effect of 128-dimensional deep features and the dimensionality reduction effect of LDA are listed in It can be seen from The reason is that most of the original 128-dimensional deep features are facial semantic information, which has little effect on the facial expression recognition that requires detailed information. Therefore, the recognition effect before dimensionality reduction is low. LDA can project the feature to the largest distance between classes according to the label, which can also be regarded as mapping the 128-dimensional deep features to the feature space that can conduct the facial expression recognition. Due to the high expressive ability of deep features, these features can better express facial expression information after dimensionality reduction. Therefore, compared with face AU features, the 128-dimensional deep features can achieve higher recognition accuracy after dimensionality reduction.We spliced local face AU features and 128-dimensional deep features containing more global semantic information to obtain the hybrid features. The facial expression recognition effect of hybrid features is listed in As listed in Combining Tables 2Principle component analysis (PCA) has been widely used as a classic method of unsupervised dimensionality reduction. Its main idea is to use the covariance matrix of the data to calculate the eigenvalues. The eigenvalues are arranged from large to small, and the feature vectors corresponding to several largest representative components (features) are used to represent the overall characteristics of the data, so as to achieve the purpose of feature dimensionality reduction. We use LDA and PCA respectively to reduce the feature dimension for different feature sets and keep the same feature dimension after dimensionality reduction. We use the dimensionality-reduced features for the training of machine learning models to test the effect on face recognition. The results are shown in Figures 6It can be seen from Figures 62The accurate recognition of facial expression recognition is inseparable from the extraction of multidimensional features, but there is often a lot of redundancy in multidimensional features. Based on the KDEF face public dataset, this study evaluates the effect of LDA feature dimensionality reduction method on facial expression recognition. First, the extraction of multitype features is performed on the face, including 35-dimensional face AU features based on artificial construction, 128-dimensional face deep features based on the FaceNet pretraining model, and 163-dimensional hybrid features. Second, LDA was used to reduce the dimensions of three different feature sets to 7 dimensions. Then, the classical machine learning methods are used to evaluate the effect of facial expression recognition before and after feature dimensionality reduction. Finally, we compare the LDA and PCA feature dimensionality reduction methods. The experimental results show that after the LDA feature dimensionality reduction, the facial expression recognition effects of the three feature sets have been improved to a certain extent, which are better than the PCA method. The dimensionality reduction of hybrid features has the best effect on improving the effect of facial expression recognition, with the macro-F1 reaching 82.90%. In addition, we found that no matter whether the feature dimensionality reduction is carried out or what machine learning method is used, the hybrid features can achieve a higher recognition accuracy than a single one. It shows that it is important to extract the multidimensional hybrid features in facial expression recognition.As this study only compares the same number of dimensions when reducing the dimension, it may need to dynamically adjust the feature dimensions for different feature sets. Hence, in the future, we will explore the effect of the feature dimensionality reduction under the optimal number of dimensions. The research on the effect of feature dimensionality reduction will be further expanded from multiple directions, such as more machine learning methods, more feature dimensionality reduction methods, and more different types of datasets.


The structure and dynamics of membranes are crucial to ensure the proper functioning of cells. There are some compounds used in therapeutics that show nonspecific interactions with membranes in addition to their specific molecular target. Among them, two compounds recently used in therapeutics against COVID-19, remdesivir and favipiravir, were subjected to molecular dynamics simulation assays. In these, we demonstrated that the compounds can spontaneously bind to model lipid membranes in the presence or absence of cholesterol. These findings correlate with the corresponding experimental results recently reported by our group. In conclusion, insertion of the compounds into the membrane is observed, with a mean position close to the phospholipid head groups. Cellular membranes are essential for life. They are responsible for preserving the homeostatic environment within the cell and maintaining crucial cellular functions . As an iMembranes are assemblies of molecules forming supramolecular structures whose properties depend on a multitude of factors such as composition, temperature, lateral and hydrostatic pressure, ion concentration, pH, and the presence of ligands such as drugs or proteins . Even smWith the emergence of the current COVID-19 pandemic, the urgent need for applicable antivirals was impressively enhanced. Since the development and approval of new drugs are time-consuming processes, a variety of already existing drugs have been tested for their potential use against coronavirus. For the latter, specific nucleoside analogues were developed as inhibitors of the virus-specific RNA-dependent RNA polymerase (RdRp), an enzyme which is also involved in the replication of the SARS-CoV-2 genome. Two existing RdRp inhibitors , remdesivir and favipiravir, were repurposed as a fast therapeutic approach for an immediate treatment of COVID-19 ,15. Remd+/Cl− ions  to represent the experimental conditions. Additionally we prepared a POPC:Cholesterol (0.8:0.2 molar ratio) mixture membrane containing a total of 128 lipid molecules with the same amount of solvent and ionic species. Initial configurations for each simulated system as well as final membrane composition details are shown in 2. The system was then allowed to relax in the NPT ensemble in 10 short phases without restrictions for a total of 5 ns using a Berendsen barostat with anisotropic scaling followed by 500 ns production dynamics at the same conditions.We prepared 1-palmitoyl-2-oleoyl-sn-glycero-3-phosphocholine (POPC) model membranes using CHARMM-GUI . A POPC Remdesivir and favipiravir parameters were obtained using a standard protocol. We calculated RESP charges using a Hartree–Fock single point calculation with 6–31G* basis set. Bond, angle, and dihedral constants, together with atom types, were assigned from other moieties in the GAFF Forcefield using Antechamber and Parmcheck modules. For modeling membrane systems in the presence of inhibitors, a membrane was first set up using the same procedure as the one for the pure membrane described above. Subsequently, the system was desolvated and a defined number of molecules of either remdesivir or favipiravir were introduced using a Packmol . In ordeWe conducted MD simulations for the 12 systems described in Methods. First, from the low concentration systems considered, the binding of each inhibitor to the membrane was followed by measuring the position of the center of mass during the simulation time, as presented in Remdesivir is, on the other hand, a bulky molecule that is not easily incorporated into the bilayer since the insertion–exit events are only infrequently observed. Remarkably, aggregation of the drug molecules was monitored in the bilayer interphase (both from the aqueous or lipidic phase), and this promotes a persistent adsorption. These observations are shared in both high and low concentration conditions. It can be concluded from the time evolution of the center of the mass of the molecules that favipiravir molecules are distributed independently throughout the simulation, whereas remdesivir molecules tend to aggregate.Once the compounds were incorporated into the membrane, we proceeded to evaluate the degree of membrane penetration, as shown in the electron density profiles in In an attempt to describe the most important intermolecular interactions and the organization of the compounds once inserted into the membrane, we proceeded to calculate the molecular nature of the closest structures to each compound throughout the simulation for both inhibitors. The results are shown in In an effort to define a preferential orientation of the inhibitors in the studied membrane structures, in 2. However, the presence of cholesterol in the membranes yielded a reduction in the area per lipid, from 59.9 to 54.0 and 55 Å2 for the presence of favipiravir and remdesivir, respectively. Order parameters represent the effect of the insertion of inhibitor molecules in the packing of the membrane lipid acyl chains. For pure POPC membranes, the presence of either remdesivir or favipiravir does not significantly decrease the order parameter in the middle region of the lipid chains. In a POPC:Chol membrane, the effect of the inhibitor presence is roughly neglectable in terms of acyl chain packing, as derived from the unaltered order parameter profile. One possible explanation for this is the fact that cholesterol molecules, when present, tend to come closer to the inhibitor molecule, moderating its effect on the acyl chain packing. To assess the effect of inserting compounds on the membrane structure, we calculated values for lipid area and order parameters, shown in The results presented in this study are accompanied by the experimental results presented in a separate paper dealing with the impact of the two drugs, remdesivir and favipiravir, on the properties of different membrane systems . As noteWe would first like to comment that we are aware that our simulations as well as the associated experiments  were perIn agreement with the experimental data, the simulations demonstrate that remdesivir and favipiravir show spontaneous penetration into the model membranes. The transition from the aqueous to the lipid phase occurs under the experimental conditions tested, and lipid embedding is also observed in the simulations in which the compounds start in the lipid phase and remain in this phase at least for the simulation time explored. The partitioning of drug molecules in lipid membranes can be experimentally estimated by octanol-water partition coefficients (logP) . Interes1H–1H MAS NOESY experiments, which allow the identification of the interaction regions of drugs within the membrane [Another parameter to consider is the degree of penetration of the explored compounds into the model membranes. As derived from the electron density profiles, the compounds studied were inserted into the membrane in the region close to the phospholipid heads, with a slightly different degree of penetration for each compound. This is in line for remdesivir with cross-relaxation rates measured in membrane . These rEven though the presence of cholesterol in the membranes has only a little effect on the final membrane insertion of both studied compounds, some structural changes regarding the presence of it in the membrane could be observed. Moreover, changes in the cholesterol–cholesterol interaction pattern were observed . NeverthWith regard to the physiological relevance, we underline that our study is not aimed to present a mechanism for the antiviral activities of the drugs to explain their efficacy. However, the experimental and theoretical data on lipid membranes may help to understand the cellular uptake mechanism and putative side effects of the molecules. With regard to the latter, a disturbing impact of the drugs on membranes might explain a triggering of apoptosis or decrease of cell viability . RegardiThe present study describes the insertion of two different antiviral drug compounds as a consequence of binding to lipid membranes in accordance with the changes exerted on a model membrane structure and dynamics. It was shown that while remdesivir incorporates into the bilayer causing a disturbance of the membrane structure, favipiravir has no indications for membrane impact. The MD simulation could confirm and well correlate with previous experimental data on an atomistic level.


A new breast imaging system capable of obtaining ultrasound and microwave scattered-field measurements with minimal or no movement of the breast between measurements has recently been reported. In this work, we describe the methodology that has been developed to generate prior information about the internal structures of the breast based on ultrasound data measured with the dual-mode system. This prior information, estimating both the geometry and complex-valued permittivity of tissues within the breast, is incorporated into the microwave inversion algorithm as a means of enhancing image quality. Several techniques to map reconstructed ultrasound speed to complex-valued relative permittivity are investigated. Quantitative images of two simplified dual-mode breast phantoms obtained using experimental data and the various forms of prior information are presented. Though preliminary, the results presented herein provide an understanding of the impacts of different forms of prior information on dual-mode reconstructions of the breast and can be used to inform future work on the subject. The use of microwaves to quantitatively image the complex-valued permittivity of the breast  into the inversion algorithm. Several past studies have investigated methods of extracting prior information from other imaging modalities ,6,7,8, ic speeds . AnotherWith these factors considered, the modality chosen to complement MWI in this work is a novel, inexpensive quantitative USI system . PreviouTo validate the new system, two dual-mode breast phantoms  were developed. A quantitative 3D ray-based US inversion algorithm was used to produce images of the acoustic speed of the phantoms . The proA system-level description of the new dual-mode system is provided in Hold the breast in a known, fixed position and geometry while imposing minimal compression on the breast;Allow the US transducers to make direct contact with its surface, via slight compression or the use of an ultrasonic coupling gel;Have favourable acoustic properties: limited attenuation and limited reflection at the skin interface so that sufficient acoustic energy penetrates the breast; andHave favourable MW properties that allow MW energy to interrogate the breast while being easily incorporated in the MWI inversion model (as prior information).As reported in ,23, the In the final design, mechanical infrastructure will allow the lowering of the MWI chamber so that the USI fixture may be fastened to the breast-support cup. For results shown in this work, data for both modalities were collected with the target held by an X-shaped plexiglass support. The US fixture locks into the X-shaped support and markings were made on the top of the MWI chamber so that the support could be placed consistently at the same location and the OI’s position could be known exactly. After MWI data were acquired, the OI and support were carefully transported to a temporary container where the US fixture was attached.The flow of data through each of the USI and MWI systems is detailed in An arbitrary waveform generator (AWG), comprised of four PC-based waveform generator cards (Signatec PXDAC4800), each with four channels. A pulsed sinusoid truncated after five periods was used to drive the US transducers at their observed resonant frequency of 1.4 MHz.A 32-channel digital oscilloscope, composed of four PC-based oscilloscope cards (GaGe Octopus 8387 CompuScope digitizer boards), each with 8 channels.A custom switch module that routes the transmitted signal from the waveform generator to the specified transducer and received signals from transducers to the corresponding oscilloscope channels.MATLAB control software that coordinates the above components, including a graphical user interface (GUI).Custom A novel, 3D-printed fixture that holds the 64 piezoelectric transducers in direct contact with the breast-support cup.The US data acquisition system consists of the following parts:The US transducers  are single-element piezoelectric transducers. Each element consists of a cylindrical lead zirconate titanate (PZT-5H) crystal, connected to a twisted pair of wires. The crystals are dipped in epoxy to provide better coupling to the background material, assumed to be water or similar. The twisted pair is surrounded by a ground shield and encased inside a waterproof coating. The usable bandwidth of the transducers is approximately 1.0–1.8 MHz.The US fixture is made up of two hollow semi-cylindrical pieces plus a semi-spherical cup, housing 64 transducers arranged in a helical pattern. Small, keyhole-shaped cutouts hold the transducers against the breast-support cup. The three-piece fixture is fastened together using plastic pin connectors and nylon screws and is easily taken apart to remove the fixture from the target after data collection. The fixture is designed to fit snugly around a flexible capped-cylindrical container that doubles as the breast-support cup and the skin layer of the breast phantom . US coupThe microwave measurement system used here is the same air-based, flat-faceted, quasi-resonant metallic chamber published in . This syThe first dual-mode phantom (Phantom 1) was a simplified breast model consisting of skin, fat, and tumour regions only . The skiThe physical properties of the breast phantoms used are approximations of the parameters that might be expected for actual tissue. Published values of acoustic speed at 1.5 MHz and relative permittivities at 1.1 GHz of real breast tissues are provided for reference in italics in The dual-mode setup described at the start of d that were inverted by CSI were obtained as,U denotes measured fields and H denotes synthetic fields modelled with the FEM forward solver [C. For the results shown in this paper, the calibration object and incident field background are the same, i.e., For the MW inversions, we utilized our FEM-CSI algorithm with prior information acquired from the US speed image ,31. Seve tumour) ,34. The One of the main contributions of this work was in finding a methodology for incorporating the US-derived prior information into the MWI algorithm. In this section, we describe our initial study of possible methods for generating prior information from the US speed images. In Segmented mapping—a k-means clustering algorithm was used to segment the sound speeds into two or more regions, e.g., fat and fibroglandular. Representative complex permittivity values for the tissues are assigned to the respective regions.Linear mapping—reconstructed sound speed values c are linearly re-scaled to ranges representative of real and imaginary parts of complex permittivity, using the following equations:Tissue-range mapping—a range of values of each physical property for each tissue type is set, based on the representative complex permittivity values. For each reconstructed value of speed, the distance from the mean of the corresponding tissue-dependent range is calculated. Complex permittivity is then assigned as the same relative distance from the mean for the given tissue type. This type of mapping is based on the tissue-dependent mapping technique described in [ribed in .To use information from an US image in FEM-CSI, the reconstructed sound speeds must be converted to complex-valued permittivities. Three different functions for mapping sound speeds to permittivities were investigated:For the results shown herein, the representative complex permittivity values for all mappings are obtained from the measured properties of the phantoms. The tissue-dependent ranges used for Phantom 1 are provided in As described in more detail in , the rayAs seen in the figures, the US reconstructions provide a fair approximation of the physical structure of the tissue regions. However, the reconstructed sound speeds do not agree exactly with the measured values. The maximum reconstructed speeds for Phantoms 1 and 2 are 1542 and 1580 m/s, respectively, whereas the maximum measured speeds of the phantoms are 1587 and 1595 m/s, respectively. Additionally, the reconstructed speeds were observed to increase with the order of polynomial basis functions used. Higher order bases were observed to produce artefacts at the boundaries of the imaging domain . Thus, tThe three techniques described in For Phantom 2, because the USI on its own is not expected to recover the tumour due to the lack of sound speed contrast with the fibroglandular region, the number of regions for the segmented mapping is kept at two. The cluster with lower speed is assigned the measured permittivity value of the fat-mimicking material and the higher speed cluster assigned that of the fibroglandular phantom material. The resulting permittivity prior for all three mapping techniques are shown in A selection of the complex permittivity maps shown in the previous section was incorporated into the FEM-CSI algorithm as an initial guess. In this section, we compare the quality of the resulting images when different types of prior information are used. As a comparison to the dual-mode results, we also provide the MWI results obtained using the FEM-CSI algorithm but without US-derived prior information. For this case, the MW measurement data obtained with just the skin and fat regions of the phantom are utilized as the incident measured field, gated in , the cloThe FEM-CSI reconstruction results for Phantom 1 obtained using four different variations of prior information are shown in Evaluation of the resulting images has, thus far, been limited to qualitative analysis. From the images in Regardless of the form of the prior information, the skin seemingly disappears in some spots and has high permittivity in other spots. From a tumour-detection point of view, this is undesirable as it would lead to false positives within the skin region. Thus, for the upcoming reconstruction results for Phantom 2, the skin region was excluded from the imaging domain.Based on the results seen for Phantom 1 and the difficulty with the segmented mapping for Phantom 2, the tissue-range mapping was selected as the technique for producing prior information for Phantom 2. Thus, the permittivity map depicted in As seen from the images in The results presented in this paper demonstrate the feasibility of a novel experimental integrated 3D dual-mode microwave-ultrasound breast imaging system. A complete workflow of taking low-resolution US speed reconstructions, converting them to complex-valued permittivity, and incorporating them as prior information for MWI has been described. Preliminary testing of this workflow was performed using two simplified breast phantoms to validate the soundness of the imaging procedure. Three different methods for generating prior information from the US reconstruction for use in the MW inversion algorithm have been investigated. Though there is room for modifications to be made to the sound speed to permittivity mappings, we conclude that the tissue-range mapping is highly beneficial due to its robustness to sound speed artefacts and the ability to map all tested tissue types to realistic permittivity values.Although not presented herein, the utility of the resulting images to enhance tumour detection was investigated in ,23 usingOur future work includes the development of more realistic dual-mode tissue-mimicking breast phantoms so that the imaging procedure can be more thoroughly evaluated. The value of constructing higher resolution US images is an open question, although such reconstructions may require a more elaborate US data acquisition fixture. Alternatively, using the same low-resolution data, the resolution of the US images could be improved by reconstructing additional US parameters, such as attenuation, or the use of other, qualitative reconstruction techniques, such as DAS ,35. The Any future clinical investigation would require some modifications to our current system. Perhaps the largest hurdle to overcome is in the design and fabrication of a suitable breast-support cup, or cups, that could be used during both MW and US data collection. To accommodate the wide possible range of patients, support cups of various shapes and sizes could be manufactured and even custom-fitted to a patient’s breast. Additionally, for both modalities, we will require a calibration object and background incident field that are good approximations of the patient’s breast. As it is unlikely that we will have access to measured data for the same breast with and without a tumour, the use of a breast phantom with physical properties that approximate the bulk-average properties of a patient’s breast could be used. This phantom may consist of the breast-support cup filled with some material that approximates the bulk MW and US properties. This would require a study of how the mismatch between the phantom properties and the true properties of the breast affects the reconstruction quality.