Particle mixing process is critical for the design and quality control of concrete and composite production. This paper develops an algorithm to simulate the high-shear mixing process of a granular flow containing a high proportion of solid particles mixed in a liquid. DEM is employed to simulate solid particle interactions; whereas SPH is implemented to simulate the liquid particles. The two-way coupling force between SPH and DEM particles is used to evaluate the solid-liquid interaction of a multi-phase flow. Using Darcy’s Law, this paper evaluates the coupling force as a function of local mixture porosity. After the model is verified by two benchmark case studies, i.e., a solid particle moving in a liquid and fluid flowing through a porous medium, this method is applied to a high shear mixing problem of two types of solid particles mixed in a viscous liquid by a four-bladed mixer. A homogeneity metric is introduced to characterize the mixing quality of the particulate mixture. The virtual experiments with the present algorithm show that adding more liquid or increasing liquid viscosity slows down the mixing process for a high solid load mix. Although the solid particles can be mixed well eventually, the liquid distribution is not homogeneous, especially when the viscosity of liquid is low. The present SPH-DEM model is versatile and suitable for virtual experiments of particle mixing process with different blades, solid particle densities and sizes, and liquid binders, and thus can expedite the design and development of concrete materials and particulate composites. Mixing of granular materials is crucial in a broad range of industrial processes, including constructional material production, advanced composite manufacture, mineral processing, plastics manufacturing, ceramic component, pharmaceutical tablets, and food products. Mix uniformity plays a critical role for the overall performance. Many experimental approaches have been employed in quality control of the mixing process, such as sampling, visual tracking, magnetic resonance imaging, rheometer measurements, etc. Among others, Khan et al.  used higMoreover, it is time-consuming to develop a new particulate composite through such a trial-error method. Therefore, numerical simulation becomes an alternative to investigate granular flow dynamics.For solid applications, Orefice and Khinast applied DEM to study the transporting process of screw conveyors with different inclinations and filling levels . And SinContinuous media models treat both solid and liquid phases as interpenetrating continua. This scheme works well for problems with small deformation, but often fails in problems with large deformation, such as mixing problems. In comparison, discrete particle methods can simulate the dynamics of particulate mixtures and are capable of dealing with moving boundary problems in the mixing process, such as the Molecular Dynamic (MD), Discrete Element Method (DEM), Smooth Particle Hydrodynamics (SPH), and Dissipative Particle Dynamics (DPD). In these methods, some specific treatments are generally required to evaluate the coupling between solid particles and liquid particles ,11,12.Other methods combine continuous media and discrete particles, such as CFD coupling DEM ,14, wherTo tackle mixing problems with an intermediate to high solid load, SPH-DEM coupling is a promising approach. Cleary  proposedIn comparison to DEM particles added to SPH liquid, some preliminary work has been conducted towards SPH liquid flow in porous media. Shao  and Zhu However, it is still challenging for the coupled SPH-DEM method to simulate particle mixing with a high solid load due to the difficulty of calculating the interaction force between SPH and DEM particles, because most previous models were built under the assumption that solid particles are fully saturated by the liquid phase. This work aims at the solid-liquid mixing problems with very high solid loads, say up to 90% in volume. Here the key issue is how to calculate the interaction between liquid and solid particles. To compute the local liquid density, solid particles are considered to be rigid without any overlap with each other. Rather than excluding solid particles by dividing porosity in the density computation , this paIn the following, the numerical algorithm and formulation of the two-way DEM-SPH coupling method are introduced first. Two benchmark studies, i.e., (1) a solid particle sedimentation among liquid and (2) liquid flow through a porous media, are presented to verify the model for the parameter calibration. Subsequently, the present method is applied to high shear particle mixing problems, and a series of parametric studies are conducted. The local average mixing index is the main criterion to evaluate the homogeneity of the mixture, which will be used to optimize the particle mixing process for homogeneous mixtures.The Smoothed Particle Hydrodynamics (SPH) has been widely used for simulating fluid flows, since it was first developed by Gingold  initiallx, SPH discretizes the domain to particles as follows:h is the Kernel length which determines the interaction domain of the kernel function; V is the material domain. In this work, Cubic Spline kernel function [h, the more the neighboring particles to be counted [h are chosen for specific problems of interest considering a balance between accuracy and efficiency.The value of x and x. The density of each particle in SPH is calculated by the mass and kernel functions as follows:In an SPH model, each particle has its properties such as position, velocity, mass, pressure and energy. Pressure and temperature are functions of density and kinetic energy, respectively. During the flow process, the mass of each particle stays constant while the density keeps changing as the flow is assumed to be weakly compressible. For any variable distance  and r=|xfunction  is imple counted . DiffereIn practice, the continuity equation is automatically satisfied due to the principle of mass conservation of each particle . Besidesi and particle j is appliedThe Tait equation has a parameter of sound speed ble flow ,39. To discosity , ν0 is tNotice that due to the approximation using discrete particles to represent a continuous flow, the modeling parameters in a particle method will not only depend on the fundamental material properties, but also change with particle size, modeling scale, and neighborhood cutoff-distance, among other factors. To obtain repeatable and reliable results, a procedure for the calibration and verification of the modeling parameters should be conducted before using the method in virtual experiments. I, respectively. DEM particles have radius, and thus can have torque. He and Bayly coupled DEM and SPH to simulate interactions of liquid with solid phase [Discrete Element Method (DEM) is a numerical method to simulate the motion and interaction of granular materials as discrete particles. The major physical law is the momentum conservation, i.e., Newton’s second law. The governing equations of DEM could be written as:id phase . In the id phase . Followiid phase . The sofAmong the existing soft-sphere models, there are various approaches proposed to solve the relation between particle overlap and interaction forces. The most common method is the linear spring-dashpot (LSD) model, in which the interaction between two particles is expressed by a normal spring and dashpot, a tangential spring and dashpot, and a torque . It coulIn the LSD modeling, there are two types of forces: normal n energy , maximumn energy , and dimn energy  to the nThe tangential spring stiffness  follows :(12)kt=2e, and the friction coefficient Although the parameters in the above force units have clear physical meaning, it is not straightforward to identify the appropriate value in the DEM simulation due to the significant simplification in material modeling, time integration, and geometric consideration. Therefore these parameters are also determined through comparing numerical results with classic contact mechanics models, such as the Hertz model, the Johnson-Kendall-Roberts (JKR) model, the Derjaguin-Muller-Toporov (DMT) model, etc., or calibrated by the experiments. For example, in many cases, the normal spring stiffness  results . Therefoi, its porosity When liquid and solid particles coexist and interact with each other, the coupling force should be considered. In this work a two-way SPH-DEM coupling method is introduced. In contrast to one-way coupling where only one kind of particles exert force on the other, the two-way coupling means that both SPH particles and DEM particles exert forces on each other. Prior to computing the coupling force, the density of SPH particles needs to be revised to prevent overlap between DEM and SPH particles. To this end, one can introduce the concept of porosity, and then the volume fraction of liquid and solid phase are specified . For an Then the density of the SPH liquid particle is calculated by the weighted sum of its neighbor particles divided by its porosity:In this way, when a DEM particle enters the neighborhood of a SPH particle, the porosity of this SPH particle decreases, and its density increased, generating repulsive forces between its original neighbor SPH or DEM particles . When these neighbor particles reach balance again, the number of SPH or DEM particles in the SPH particle’s region is decreased, thus making room for the entering DEM particle and no particles are overlapped during this process.However, at mixing cases where the solid load is as high as 80% to 90%, the porosity of SPH particles is inevitably low, which makes the computation of liquid density sensitive to the changes of its position and easy to become unstable. To solve this problem, a new method to compute the SPH particle’s density is proposed as below:As for the coupling forces between DEM and SPH particles, one common approach is to apply buoyancy or pressure gradient forces and drag forces on DEM particles, and then apply the counteractive forces on SPH particles to attain two-way coupling. This approach is based on the assumption that solid particles “float” in liquid, in which liquid take the majority of the space. However, when the solid load is high, the assumption that solid particles are fully saturated by liquid particles does not hold anymore. In this work, a more fundamental formulation is derived and the coupling forces are still comprised of two parts, i.e., pressure gradient force and viscous force. For the pressure gradient force, the formula is the same as the SPH pressure gradient force, i.e., treating part of the solid-liquid interaction as SPH pressure gradient forces. Pressure gradient forces make SPH and DEM particles stable and do not overlap, while the essential coupling mechanism is the viscous force.v is the velocity of the flow, k is the permeability of the porous media, In flows among porous media, when the solid portion of the porous media is high, its macroscopic behavior can be described by Darcy’s Law :(16)v=kμeriments . SPH is k is a function of porosity, the viscous force between DEM and SPH particles is a function of porosity. When the liquid load is high enough in the simulation, the interactive viscous force between DEM and SPH particles will be close to the viscous force between the SPH particles. Depending on the porosity of the target particles, the coupling viscous force will be multiplied by a coefficient, which is a function of porosity, and satisfies the classical Darcy’s Law [i, its i and j that calculated using Equation  should be at least five times of the average particle diameter  so that http://lammps.sandia.gov/index.html, version: 16 March 2018). When observing the lack of available sources to conduct particle dynamics simulation with various particle based potentials, such as SPH, DEM, dissipative particle dynamics (DPD) [The simulation in this work is done by software package Particle Dynamics Parallel Simulator (PDPS), developed in the Pao Sustainable Engineering and Materials Laboratory (Pao Lab) of Columbia University. It was inspired and designed based on the structure from the open source molecular dynamics software LAMMPS (cs (DPD) , etc., wPDPS is written in C++, and it uses Message Passing Interface (MPI) to conduct the parallel computing in a distributed memory environment. It can be run on both a single processor or multiple processors machine which can compile C++ and support the MPI library. The parallelism is fulfilled by using the domain decomposition technique, which has been demonstrated as one of the most efficient parallel algorithm towards molecular dynamics simulation . A schemn is the total number of particles simulated, and k is the average number of neighbor particles that each particle has, typically around 100. A flow chart is provided in Similarly to LAMMPS, the architecture of PDPS includes the following modules: (1) initialize the computational model; (2) build the list of neighbor particles; (3) calculate pairwise forces between particles based on the neighbor list (4) time integration  to update particle position and velocity; (5) repeat step (2) if simulation is not finished. Pairwise force computation is the most time consuming part, thus the time complexity of the particle dynamics simulation problem is One major concern of particle dynamics methods is how to choose simulation parameters correctly. In this work, the parameters in simulation is chosen from classical benchmark studies. Parameters used in the SPH-DEM model are chosen to match some classical solid-liquid two phase flow problems, and these parameters will be applied to solid-liquid mixing problems.In this case, a single DEM solid particle falling in a fluid of many SPH particles is investigated. This test is the first step to validate the feasibility of the DEM-SPH coupling method. As shown in h is the smoothing length of SPH particles, and it is considered as the radius of SPH particles. The smoothing length is defined the same way in the following case studies. Since the Re number in this case is less than 1, it could be considered as a Stokes flow.Properties of DEM and SPH particles are listed in Based on the Stokes flow equation, the velocity of this single DEM particle could be expressed as:Since the coupling forces between DEM and SPH particles have a range of interaction domain rather than stay on a distinct radius, a multiplier is applied to the viscosity forces between DEM and SPH particles to get the correct Stokes flow velocity. It is found when the multiplier is around 3.08, the Stokes flow could be reproduced as shown in In this example for sedimentation of a single DEM particle among SPH liquid particles, the coupling force can effectively capture the physics of the interaction between liquid and solid particles. Notice that the surface toughness and surface tension of the particle may play an important role for the particle sedimentation , this beThe second case study evaluates the effectiveness of the present method on a more complicated scenario, liquid flowing through a porous media. In this case a U-tube with a square width of 10 m and a horizontal length of 60 m is generated, as shown in The properties of SPH and DEM particles are listed in C is a problem dependent parameter to be determined. As shown in As the liquid flowing from one side of the U-tube to the other side, the flow speed and the height difference between the two sides depend on the porosity of the porous media and viscosity of the liquid. Based on previous studies , the heiAfter the successful demonstration of the present method in the two case studies, it is applied to the simulation of a solid-liquid mixing process in a high shear mixer with four blades. As shown in In the simulation conducted below, two types of solid particles and one type of liquid particles are employed to investigate the mixing performance of solid-liquid two-phase flow. The properties of the two types of solid particles are the same except color, which is used to demonstrate the mixing process. Their properties are listed in Prior to mixing, the solid particles are classified into two groups , with each group filling half side of the container, as shown in Then liquid particles are generated at the top of these solid particles. After these particles reach stable, the blades start to rotate with a constant acceleration rate until a given velocity is reached. Then the two groups of solid particles together with liquid particles are mixed towards homogeneous distribution until the rotation of the blades stopped. The liquid acts like a binder to make two groups of solid particles bound together. It is similar to the cohesive effects in liquid bridge model, but here SPH liquid particles could deal with a high liquid portion (>10%). In the following parts, the mixing process is analyzed in detail, and the effects of different variables such as the amount of liquid content, viscosity of liquid on the mixing performance are investigated.Since at the beginning two kinds of solid particles are placed in symmetry, the radial distribution and height distribution of the two types of solid particles are expected to be comparable after certain time of mixing as shown in In addition to the top view, the distribution of liquid at different height of the cylindrical container is expressed in Notice that while the solid particles are distributed homogeneously along the vertical direction of the cylinder, the liquid mainly stays in the middle range. Therefore, the liquid is unable to be mixed well vertically within the solid particles in the current configuration. In In this part, the effect of liquid amount on mixing performance is studied. To test the mixing quality, local average mixing index is adopted here to examine the homogeneity of the particles with a cubic RVE of size 30 mm × 30 mm × 40 mm. In the following case studies, the mixing index is defined the same way. In this part, the effect of liquid amount on mixing performance is studied and the corresponding results are shown in Top view of the distribution of liquid particles is shown in The viscosity of liquid has a significant effect on the mixing performance. Three different kinematic viscosity 0.04 mA high viscosity inhabits the mixing process. When the viscosity of liquid is low, its effect on the mixing performance is not obvious, but when the viscosity of liquid is high, the viscous force between liquid and solid is also high. The liquid particles act like binders and bind solid particles together as local aggregates, which makes it hard to mix the solid particles completely. It is found that when the viscosity coefficient is small (0.004 and 0.2), the effect of viscosity on mixing index is not material. When the viscosity coefficient is 1 the mixing index curve differs significantly from the other two. The specific viscosity coefficient value for the mixing index curve to change dramatically should be explored in future. At the initial period, the mixing index is the same for all three cases, it is because at the initial stage the rotation speed of the blades is low and viscosity has not started to show its impact. In In this part, a group of figures of liquid particles distribution are shown in h has the physical meaning of constructing list for neighboring particles. Robinson and Rosswag pointed out that [d is the dimensionality of the simulation problem; h could be simplified as Equation h has theEquation ,(31)h=ηh = [r is the radius of the solid particle. Because the estimation is based on the solid particles occupy entire domain, and the h is constant during simulation, h beyond above range is expected. To investigate how smoothing length h influence the mixing index, a study is made with three different smoothing length, h = [h will cause some fluctuations during the mixing process, however, it does not have strong effect on the final state. Therefore, one can conclude that under rational estimation with density, mass and DEM particle radius, the smoothing length h can be selected without affecting results.Based on the situation that the solid load is high, therefore one can estimate by assigning the solid particles full influence, which yields Adding more liquid can slower down the mixing process for a high solid load mix, and increasing the viscosity of the liquid will slow down the mixing process tremendously.When visicosity of liquid reduces, although the solid particles can be mixed well eventually, the liquid distribution is not homogeneous, but concentrates at the edge due to the centrifugal effect.A smoothing length at 1.5–2 times of the particle radius can provide convergent results of mixing index for virtual experiments.The two-way SPH-DEM coupling method that covers the full spectrum of solid loads, especially for a high solid load, is presented to study high shear particle mixing problems. In this coupling scheme, a porosity is introduced for each SPH particle to calculate its density. The coupling force between SPH and DEM particles includes two parts: SPH pressure gradient equation is implemented to compute the pressure gradient force between DEM and SPH particles; whereas the viscous force is related to particles’ porosity by Darcy’s Law. In virtual experiments, two types of solid particles and one type of liquid particles are mixed in a four-bladed mixer. The mixing index measured by local average method is the main criterion to quantify the quality of the mixing status. The mixing processes with different proportions and viscosity values of the liquid are investigated. The virtual experiments with the present algorithm show the following aspects of the particle mixing process with high solid loads:Overall, the present SPH-DEM coupling method is stable and reasonable when the solid load is high, and this method could be applied to many kinds of solid mixed with a liquid. Particularly, for concrete production, fine aggregate, cement, and admixture powders are mixing with water. To obtain homogeneous mixture, as the physical experiments are expensive and time-consuming, replacing some physical tests with virtual experiments on computer may expedite the design process of concrete. Although the present method requires the calibration of parameters, it can reproduce the physical experiments on computer and advance the understanding of particle mixing. The model can be extended to other applications of particle systems.The present method can be extended to virtual experiments of particle mixing process with different mixer blades, solid particle densities and sizes, and liquid binder. One can use the method to formulate the mixing process and thus significantly expedite the material development cycles.


Traffic speed prediction plays an important role in intelligent transportation systems, and many approaches have been proposed over recent decades. In recent years, methods using graph convolutional networks (GCNs) have been more promising, which can extract the spatiality of traffic networks and achieve a better prediction performance than others. However, these methods only use inaccurate historical data of traffic speed to forecast, which decreases the prediction accuracy to a certain degree. Moreover, they ignore the influence of dynamic traffic on spatial relationships and merely consider the static spatial dependency. In this paper, we present a novel graph convolutional network model called FSTGCN to solve these problems, where the model adopts the full convolutional structure and avoids repeated iterations. Specifically, because traffic flow has a mapping relationship with traffic speed and its values are more exact, we fused historical traffic flow data into the forecasting model in order to reduce the prediction error. Meanwhile, we analyzed the covariance relationship of the traffic flow between road segments and designed the dynamic adjacency matrix, which can capture the dynamic spatial correlation of the traffic network. Lastly, we conducted experiments on two real-world datasets and prove that our model can outperform state-of-the-art traffic speed prediction. In the field of roadway transportation, traffic speed is an important and fundamental parameter, and it can reflect the dynamic running of road traffic. In practical scenarios, if the traffic speed can be known in advance, it can be used to optimize traffic control effectively and improve traffic efficiency greatly. With accurate values of traffic speed known ahead of time, traffic management authorities can not only reasonably adjust the signal timing of road intersections to guide more vehicles in urban traffic but can also control the number of vehicles at entrances to alleviate the traffic flow of expressways. Nowadays, traffic speed prediction has become an indispensable part of intelligent transportation systems (ITSs) , and becS, location B is the nearest neighbor and has a greater impact on location S, but when there is a traffic jam caused by a collision in location B, the impact of locations A and C on S will be greater than location B as time goes on. Thus, the spatial–temporal correlation of traffic data is complex and highly dynamic, and it is a great challenge to extract the spatial–temporal correlation from traffic data.Traffic speed prediction is a typical problem associated with spatial–temporal data, especially in expressway or highway scenarios. Expressways are relatively closed and have a high-speed traffic flow, and their traffic data are generally collected from numerous detectors scattered in different locations. The change in traffic speed is not only affected by the vehicles in the target location but is also related to the vehicles converging from the adjacent locations, and the impact of each neighboring location also changes as time goes by. As shown in The earliest methods of traffic speed prediction are based on the traditional statistical model, such as historical average (HA), vector auto-regressive (VAR)  and autoWith the rapid development of data processing techniques, data-driven methods have drawn more attention. Machine learning methods can extract rules and features from a large amount of traffic data. The k-nearest neighbor (KNN) method was first applied to traffic forecasting in  and imprIn order to capture both temporal features and spatial dependences from stochastic traffic data, some studies have improved in this area. Lv et al.  proposedRecently, spectral graph theory has realized the graph convolution operation for graph-structured data. A general graph convolution framework was firstly proposed by Bruna et al.  and furtTo overcome the problems mentioned above, we referred to the full convolutional structure proposed in  and desiWe present a novel deep leaning model called FSTGCN to forecast the traffic speed. In this model, we adopt the full convolutional structure to capture spatial dependencies and temporal features and integrate the data of traffic flow and speed together to forecast the future traffic speed.We design a dynamic adjacency matrix to extract dynamic spatial correlations effectively. We first encode the dynamics of every two locations by their covariance of the traffic flow data and then combine with the static spatial adjacency obtained from the distance.We conduct numerous experiments on real traffic datasets and verify the effectiveness of the model.To summarize, the main contributions of this paper are as follows:The rest of this paper is organized as follows. In this section, we introduce the definition and the problem statement involving traffic speed prediction. Definition 1(Traffic Networks).In this paper, we define the traffic network as an undirected graph, where  is the edge collection of the road network,  is the node collection of the road network, and , where  is the number of nodes, and each node represents a detector on the road segment. The adjacency matrix  is used to represent the connection between nodes.Definition 2 (Feature Matrix).The feature matrices  and  represent the historical data of traffic flow and speed, respectively. and  denote the traffic flow and speed at all road segments, respectively. and  denote the traffic flow and speed of the -th road segment over past  time slices, respectively.Problem 1.Given traffic network and feature matrices and , the model  is trained with the historical traffic data to predict the future traffic speed, as shown in the following.where  denotes the predicted traffic speed of all road segments over the next time slice, and denotes the predicted traffic speed of all road segments at timestampAs shown in The core of graph convolution is Laplacian eigendecomposition, and the Laplacian is constructed by the degree matrix and adjacency matrix. In the road traffic field, many GCNs use the spatial connectivity to represent the nodes’ adjacency, and some GCNs adopt the exponential distance to denote the spatial correlation. All these methods only consider the static spatiality of road segments, but as shown in The road network is regarded as an undirected graph, and each node represents a detector on the road segment. The traditional convolution cannot directly act on the non-Euclidean road network, and spectral graph theory realizes the graph convolution operation by converting the convolution in the space domain to the product in the frequency domain. Bruna et al.  firstly The adjacency matrix in traditional graph convolution mainly has two variants, as shown in the following, and it uses the static road topology information to represent the spatiality of road segments.Neighbor adjacency matrix: The adjacency matrix is defined according to the connectivity of neighboring road segments. This method determines whether the road segment is connected with the neighbors or not, which is as followsDistance adjacency matrix: This method uses the exponential distance to analyze the spatial correlation between road segments [segments . These tAs shown in Dynamic adjacency matrix: When the traffic flow of two road segments changes in a similar way, this can be regarded as adjacency. Thus, we can use the covariance of the traffic flow to calculate the dynamic spatial correlation between road segments, and the covariance adjacency matrix is as followsThen, considering the static spatiality, the dynamic adjacency matrix is designed as the Hadamard product of the distance adjacency matrix and the covariance adjacency matrix, which is as followsAs shown in (6), the exponential distance can decide the static spatial adjacency between two nodes, and the covariance of the traffic flow can determine their dynamic spatial adjacency. Thus, if both their covariance and distance correlation are large, the two nodes will influence each other to a large extent.When using neural networks, many researchers only apply historical data of traffic speed and capture valuable information to forecast the traffic speed in the future by training and testing the model. Several studies  by using an extra ST-Conv block and output the forecasting values of the traffic speed with the output layer. The formulas areWhen training the model, the root mean square error is employed as a loss function to measure the difference between the predicted result Furthermore, the Adam optimizer is applied, and residual connections are implemented among stacked temporal convolutional layers. We evaluated our model on two real datasets, PeMSD4 and PeMSD8. Our traffic datasets were collected by the California Transportation Agencies  Performance Measurement System (PeMS). These traffic data contain traffic flow, average lane occupancy and average speed. The details of the datasets are shown in The two datasets set the time interval of traffic data to 5 min. Therefore, each node contains 288 data per day. We used the Z-score method to normalize the data to allow the mean of the original data to be 0 and the variance to be 1. In addition, the first 60% of the data were used as the training set, then 20% were used as the verification set and the remaining were used as testing.The model FSTGCN was implemented based on the Pytorch framework. The size of all graph convolution kernels and temporal convolution kernels in the ST-Conv blocks was set to 3, i.e., Mean Absolute Error (MAE):Root Mean Square Error (RMSE):Mean Absolute Percentage Error (MAPE):We used three metrics to evaluate the prediction performance of the FSTGCN model:Historical average (HA), which uses the average traffic information in the historical periods as the prediction.Vector auto-regressive (VAR), which is the multi-variable extension of the auto-regressive model which can model the correlation between nodes.Support vector regression (SVR), which is one type of machine learning method, and a linear kernel function was used in the experiments.Long short-term memory (LSTM), which is an extension of RNN and has an input gate, a forget gate and an output gate to deal with the long-term dependency and gradient vanishing and explosion problems.STGCN, which applies the full convolutional structure  to analyASTGCN, an improved scheme  of STGCNThe proposed FSTGCN method was compared to several traditional statistics-based methods and recently proposed GNN-based models, which include the following:All experiments were trained and tested on the same Linux server (CPU: Intel (R) Xeon (R) Silver 4110 CPU @ 2.10GHz, GPU: NVIDIA Corporation GM204GL (Quadro M4000)) with the existing published implementation.To guarantee the fairness of the experiments, all approaches were trained by the Adam optimizer, which updates the parameters with a gradient descent algorithm. Additionally, the batch size and the learning rate were set as 50 and 0.001, respectively. All the experiments used historical 60 min data as input to predict the content of the next 60 min.Although the performance of the machine learning method (SVR) is beyond expectation, due to the accumulation of errors and the lack of spatial–temporal analysis, the errors increase greatly when the prediction range is expanded. The LSTM model only considers the temporal correlation of traffic data, and its prediction results are not ideal, sometimes even not better than the traditional statistical prediction methods. Comparing with other methods, GCN models obviously outperform, and this proves the effectiveness of spatial graph convolution analysis in the prediction. Meanwhile, our model, FSTGCN, is also better than other GCN models, and this verifies the advantage of our improvement by fusing with the data of the traffic flow.In particular, in order to verify the advantage of our improvement by fusing with the data of the traffic flow, we designed a new experiment to further illustrate this. In this part, we added Gaussian noise generated by Python’s random distribution function to test sets PeMSD4 and PeMSD8 to enhance the inaccuracy of the speed. As shown in In order to evaluate the importance of the predicted traffic flow, we deleted module A from the proposed model and designed FSTGCN (1), which does not fuse the traffic flow data into the input of module B. The results of the experiments on the two datasets are shown in Except for the predicted traffic flow in module A, the dynamic adjacency matrix fusing with the traffic flow data also benefits the prediction accuracy of the traffic speed. As shown in In this paper, we proposed a novel deep learning model of traffic speed prediction: fusing spatial–temporal graph convolutional network (FSTGCN). This model uses the full convolutional structure as its fundamental framework, which applies graph convolution and temporal gated convolution to capture the temporal and spatial characteristics of traffic data, respectively. Then, considering the accuracy of the traffic flow data, we fused it into the model by using an ST-Conv block and concatenated the predicted traffic flow with the historical traffic speed as the input of the forecasting module, thereby correcting the inaccuracy in traffic speed prediction to a certain degree. Meanwhile, we fused the traffic flow data into the dynamic adjacency matrix. We adopted the covariance of the traffic flow data to represent the nodes’ dynamic spatial features, combined the covariance with the static distance and designed the dynamic adjacency matrix. Comparing with other matrices, the dynamic adjacency matrix can reflect the nodes’ adjacency relationship more realistically. Lastly, experiments on two real-world datasets demonstrated that our model, FSTGCN, can outperform other models for traffic speed prediction and verified the effectiveness of our improvements.In the future, we will further optimize the network structure of the model through an attention mechanism, POI data mining and other strategies to further improve the prediction ability of the model.


Nanowires typically grow along their longitudinal axis, and the long-range order among wires sustains only when a template exists. Here, we report an unprecedented electrochemical growth of ordered metallic nanowire arrays from an ultrathin electrolyte layer, which is achieved by solidifying the electrolyte solution below the freezing temperature. The thickness of the electrodeposit is instantaneously tunable by the applied electric pulses, leading to parallel ridges on webbed film without using any template. An array of metallic nanowires with desired separation and width determined by the applied pulses is formed on the substrate with arbitrary surface patterns by etching away the webbed film thereafter. This work demonstrates a previously unrecognized fabrication strategy that bridges the gap of top-down lithography and bottom-up self-organization in making ordered metallic nanowire arrays over a large area with low cost. We present a method of growing ordered nanowire arrays, bridging the gap of top-down lithography and bottom-up self-organization. Fabrication of nanostructures can be categorized into top-down and bottom-up approaches  sandwiched by two rigid boundaries with two straight electrodes, as illustrated in 4 aqueous solution. In the freezing process, due to the partitioning effect, CoSO4 is partially expelled from the ice, forming an ultrathin, highly concentrated electrolyte layer between ice and substrate when the equilibrium is reached. Our electrochemical growth takes place in this ultrathin layer . Meanwhile, a homogeneous cobalt thin film is formed if a constant voltage is applied, as shown in 2 (see fig. S2). By applying a square electric pulse superimposed on a constant voltage, periodic ridges separated by webbed films are obtained  is applied to detect the stray magnetic field of the stereo magnetic nanowire array. One may identify head-to-head and tail-to-tail domain walls on the nanowires on the top surface of the trenches , which mThe formation of ridges at the falling edge of the applied voltage pulse is a universal phenomenon. It does not depend on the shape of the electric pulses. As illustrated in fig. S3, triangular negative pulses with different falling times and amplitudes are applied. The height of the ridge increases with the falling times and amplitudes accordingly.Two effects are essential for the unique growth phenomena presented here. One is that nucleation of metal crystallite occurs preferentially at the concave corner of the substrate and the metal deposit, which guarantees the metal deposits to develop laterally on the surface of the substrate. This effect is responsible for the lateral growth of electrodeposits on the substrate, leading to many other patterns with long-range order .It should be pointed out that the interfacial growth and the cation transfer are controlled by the Laplacian fields, which are self-organized processes. In our unique system, the interfacial growth, cation transfer, and electrolyte solidification are modulated by the applied electric pulses and controlled macroscopically. In this sense, it is beyond conventional self-organization. It is an organized growth controlled by external electric pulses. We suggest that the electrochemical growth of stereo metallic nanowire arrays presented here could be promising to solve the tremendous challenges in fabricating the microstructures for racetrack memory, the high-performance, solid-state, nonvolatile memory device using a spin-polarized electric current to drive magnetic domains along magnetic nanowires, and hence realizing much smaller physical device dimensions than comparable solid-state memory devices  in deionized water . The initial concentration of CoSO4 aqueous electrolyte is 0.03 M. Forty microliters of electrolyte solution is sandwiched by two rigid boundaries made of a microscope cover glass and a polished silicon wafer. Two parallel, straight cobalt wires  are used as the cathode and anode. The horizontal separation of the electrodes is fixed as 6 mm. This electrodeposition cell is placed in a chamber made of copper. The temperature of the chamber is controlled by a thermostat circulator . The details of the chamber design can be found in  in the thermostat circulator is set as −1°C. Because of the system’s heat loss, the temperature of the thermal chamber eventually stabilizes at about −0.1°C when the room temperature is 20°C. An electric current is applied to the Peltier element to stimulate nucleation of ice of electrolyte. By carefully controlling the electric current applied on the Peltier element, nucleation and melting of nuclei can be controlled. By repeating this process, we keep only one nucleus of the electrolyte ice in the view field of the microscope. After that, the chamber temperature is slowly decreased  to the targeting temperature, and the ice nucleus grows stably. Eventually, a flat, uniform single-crystalline ice of electrolyte is generated. When the equilibrium is reached, an ultrathin electrolyte is trapped between the ice and the substrates because of the partitioning effect in the solidification process. We carry out the electrodeposition in this ultrathin electrolyte layer. The whole growing process is monitored with an optical microscope.+ ion beam etching with the etching rate of the cobalt element calibrated in advance .We use a signal generator  to apply a specific voltage pattern across the electrodes. The applied voltage varies from a constant voltage or a constant voltage superposing with a pulsed voltage. The shape and frequency of the pulses can be programmed. Usually, the electrodeposit nucleates on a silicon or glass surface. The electrodeposit grows from the cathode toward the anode in the ultrathin electrolyte between the ice and the substrates. After the electrodeposition, the sample is rinsed with deionized water and dried in a vacuum chamber for further analysis. To remove the webbed film between the nanowires, we apply Ar4 and SF6 gases. Meanwhile, C4F8 is introduced to provide fluorocarbon as polymer coating of the entire surface of the silicon as a passivation layer. The fluorocarbon layer coated on the horizontal surfaces is immediately sputtered away by the vertical collision of the ions. However, the fluorocarbon layer coated on the sidewall survives physical etching and can only be slowly etched by the chemical reaction. The manifestation of sidewall roughness is a direct consequence of the alternating etching and sidewall passivation cycles.To fabricate the stereo standing U-shaped nanowires, we introduce silicon grating with a high–aspect ratio surface topography as the substrate for electrodeposition. The silicon grating is manufactured by the Bosch gas-switching technique  and multimode AFM in tapping mode . The magnetic state of the nanowire array is characterized by the MFM mode with a lift height of 40 nm. The magnetization of the nanowires is performed by homemade Helmholtz coils, which generate a maximum magnetic field of 0.6 T.


The measurement accuracy and reliability of thermography is largely limited by a relatively low spatial-resolution of infrared (IR) cameras in comparison to digital cameras. Using a high-end IR camera to achieve high spatial-resolution can be costly or sometimes infeasible due to the high sample rate required. Therefore, there is a strong demand to improve the quality of IR images, particularly on edges, without upgrading the hardware in the context of surveillance and industrial inspection systems. This paper proposes a novel Conditional Generative Adversarial Networks (CGAN)-based framework to enhance IR edges by learning high-frequency features from corresponding visual images. A dual-discriminator, focusing on edge and content/background, is introduced to guide the cross imaging modality learning procedure of the U-Net generator in high and low frequencies respectively. Results demonstrate that the proposed framework can effectively enhance barely visible edges in IR images without introducing artefacts, meanwhile the content information is well preserved. Different from most similar studies, this method only requires IR images for testing, which will increase the applicability of some scenarios where only one imaging modality is available, such as active thermography. Infrared (IR) is a kind of electromagnetic radiation with a longer wavelength than that of visible light. Infrared thermography has been widely used in different fields, such as monitoring , medicinAlthough significant progress has been achieved in IR imaging, the spatial resolution is still one of the major limiting factors and bottlenecks for industrial thermography applications, mainly due to the high-cost of sensors. Typically, the pixel dimension of thermography is Classic image processing methods dominated the edge enhancement for IR images before 2013. Silveman  presenteDeep learning approaches, particularly Convolutional Neural Network (CNN)-based and Generative Adversarial Networks (GAN)-based methods, have recently shown explosive popularity due to their superior performance in the enhancement of visible-spectrum images and IR images. Choi et al.  proposedMost of the aforementioned deep learning based methods require the associated RGB information to enhance the targeted IR images in the testing stage, which limits their applications on some scenarios where the RGB camera is not available, for example, active thermography in NDT. Inspired by ,34 for iCompared with the existing IR edge/detail enhancement approaches mentioned above, the proposed framework has the following novelties. Firstly, it investigates a CIMEL strategy which delivers enhanced edges for IR images based on the correspondence information from digital images. The framework directly considers the edge information in the visible spectrum with the correspondence infrared occasion into the downstream training of GAN, which is an essential and critical feature for achieving the desired performance. Secondly, the network is trained by taking into account a pair of carefully designed discriminators and several loss functions. This strategy allows the learning procedure of the visual edge information to be explicitly evaluated within another domain. Thirdly, our proposed framework provides a highly dynamic learning mechanism which adopts visible and infrared images for training while accepting infrared images only for testing. Taking the active thermography, for example, during the model training process, the RGB camera can be used to collect data along with the thermal camera to inspect simulation samples , where the physical boundary of artificial defect/damage can be measured using the RGB camera. During the model testing or real applications, the RGB camera is not required anymore and the only required input is thermal images. Then the pre-trained model can be used to better estimate the physical boundary of real damage.The overall scheme of the proposed framework is illustrated in Inspired by the image-to-image translation, this study aims to acquire certain information that establishes the specified relationship of attributes  between different imaging modalities. Through combining this knowledge/information with a learning procedure, translated from visual images and applied on IR images, we introduce a dual-discriminator CGAN based framework that iteratively enhances IR edge by inferring from the visual domain. The main challenges of such a learning approach are presented as follows. Firstly, no extra edge information should be fabricated into the enhanced IR image; Secondly, the content information (low-frequency) of the enhanced IR image should be consistent with the original one while the exclusive information in the visual image should not be transferred; Thirdly, the environmental noise should not be amplified through the learning platform.x denotes the pixel coordinate in the IR image, In order to overcome the above challenges, we propose a CIMEL framework, which can be expressed as:Denoising and edge enhancement is a pair of contradictions. It is unconscionable to directly enhance IR images with background noise since edge and noise information are both in the high-frequency domain. In order to extract the ideal edge information, Gaussian Blurring (GB) is used for denoising ahead of edge detection to avoid amplifying the environmental noise. It should be noted that, ideally, the denoising method should be chosen appropriately according to the selection of the following edge detection method.Edge knowledge is a significantly important factor in this framework, which defines the knowledge to be learned. The higher quality of edge extraction means better learning target, which leads to better enhancement effect. In this study, the edge information from the visual image is detected through a deep learning-based method, Holistically-Nested Edge Detection  (HED). Ha and b are parameters describing the weight of two input images, which were chosen as 0.6 and 0.4 respectively in this study to reach the optimal visualisation; c is a constant which was set to 0 in this study. It should be noted that the selection of these parameters only affects the visualisation of enhanced images, but not the quantitative analysis in this paper.The raw IR image and edge information from the visual image are fused and then serve as one of the inputs of the edge/high-frequency discriminator, which guides the CIMEL process. The fusion expression applied is written as:To enhance IR edges through the information derived from the visual domain, we propose a dual-discriminator CGAN to achieve the aim of CIMEL. A U-Net  “links” x standards the image coordinate, x, c represents low-pass filter whilst s denotes the range filter.The basic idea of BF is to consider both spatial and similarity information of the image to be filtered which can be considered as Equation :(3)h(x)BF has been proven to have a superior performance to reduce high-frequency noise meanwhile preserving the true edges. Therefore, it is an appropriate filter to ensure the consistency of low-frequency information with the original IR image that usually has relatively less high-frequency noise. A BF is applied within the framework to post-process the image produced by the generator to ensure the consistency of low-frequency information with the original IR image through the guidance of the content/low-frequency discriminator.A sharpening operation is also applied for the generated image from the generator to assist with emphasizing edge/high-frequency information within the CIMEL procedure. However, it should be noted that the optimal sharpening operation is subjective to the purpose of the enhancement. We applied an arbitrary linear filter to the image and the following sharpening kernel was utilized in this work:Intrinsically, we prefer the generator to enhance edge/high-frequency information more than content/low-frequency information without amplifying the noise signals. Therefore, two knowledge extractors (BF and sharing operations) at the downstream of the generator play an essential role in transferring different frequency information from the generator to the discriminators during the learning process. In order to achieve convergence effectiveness, we normalize the intensity value of the input image (the raw IR image) to As shown in Due to relatively low contrast and blurred details, as typical characteristics of IR images, one discriminator is difficult to enhance edge regions adaptively while preserving the content texture globally. On the other hand, in addition to a suitable post-denoising operation, edges and content can be conveniently separated from each other within the frequency domain. In CIMEL procedure in two different frequency domains respectively, both adopting PatchGAN  for realIntuitively, we utilize the same structure with different loss functions for the two discriminators, which guides the generator to synthesize the edge-enhanced IR image by tackling low and high frequencies information within the same scene in different ways. The Mean Square Error (MSE) loss is used for the low-frequency discriminator while the negative log-likelihood (NLL) loss is used in the high-frequency discriminator. Furthermore, initial thresholds (2.3 for HED and 2.0 for LoG) are selected for both discriminators during each step of the generator training process. Meanwhile, the proposed framework has an iterative serial structure where the generator’s output is judged by the content discriminator The loss function of the framework can be described as:The loss function of In the early stage of model training, if the error is large, MSE will penalize the large error and the effect will be more significant. However, when the error in the subsequent training phase is small, MSE is not an appropriate choice. Therefore, we use the MSE loss in the low-frequency discriminator initially, and in the next high-frequency discriminator, the NLL loss is used to ensure the final accuracy of the model.The result of comparing the influence of the two loss functions for two different discriminators is shown in In order to achieve the novel ability of CIMEL, images paired with the correspondence occasion are essential for the training procedure of the framework. We employed all the video sets within the INO video analytics dataset  to demonThe framework was implemented in Pytorch using a batch size of 1. The filter weights of each layer within the framework were initialized with a Gaussian initializer with zero mean and standard deviation of 0.02, and disabled the bias. The ADAM optimizer was used with default parameters and a fixed learning rate The results of nine videos using the HED edge detection algorithm are demonstrated in (1)Edges which are invisible in the IR spectrum but visible in the corresponding visual image. As demonstrated by the yellow rectangle in the first row of (2)Edges which are weak in the IR spectrum but strong in the corresponding visual image. For the text on the truck body (‘FedEx’), illustrated by the red rectangle of (3)Edges which are strong in both IR and visual images. It can be clearly observed that such edges, indicated by the green rectangles in In order to specifically depict the edge enhancement effect of the CIMEL framework, edges are divided into four categories for discussion:SSIM), Peak signal-to-noise ratio (PSNR) and Recall were employed to measure the similarity between the edges in the original IR image and visual image, and the similarity between the edges in the enhanced IR image and visual image. PSNR can be calculated by:I is the edge of the original or enhanced IR image and K is the edge of the visual image. SSIM can be calculated by:To quantitatively evaluate the performance, Structural Similarity Index Measure  and Canny, are shown in The detailed statistical results can be found in Addressing the strong demand of thermal image enhancement, this paper reports a novel Conditional Generative Adversarial Networks (CGAN) based approach to achieve this aim by inferring edges in thermal images from edges in visual images. To treat the high and low frequencies information separately, we introduced a dual-discriminator focusing on edges and content/background respectively. With such a model, edges of thermal images can be enhanced without the inputs of visual images in the testing stage. The qualitative results demonstrated that the proposed method can effectively enhance weak edges in IR images that are strong in visual images. Meanwhile, edges that are strong in both IR and visual images are preserved. The artefact edges, which should not appear in IR images, are compressed.It should be noted that CIMEL is not a fusion method as only one imaging modality is required during the testing stage. It is an attempt to enhance features, which are weak in one imaging modality due to the physical limitation through learning from these features, which are strong in another imaging modality. This is the main novelty of this research. The direct application of the proposed method is active thermography in NDT. Active thermography detects the surface and subsurface defects based on the temperature decay profile, where the accuracy of defect measurement is affected by the low spatial resolution of IR cameras and blurred boundaries caused by heat conduction. The proposed CIMEL technique can learn the surface defect measurement using digital cameras which usually have a much higher spatial resolution and produce sharper edges, to enhance the defect measurement accuracy of thermography. Additionally, the proposed solution can be extended to other applications, such as edge enhancement for digital images benefiting from the corresponding thermal images. This will be particularly useful for human detection and tracking under insufficient illumination in the surveillance application. The methodology for such an application is almost the same and the only difference is that the feeds of digital images and thermal images should be swapped. This paper is a proof-of-concept and the full exploration of this technique requires further study.This produced model is limited by the limited number of videos in the dataset. Although there is no overlap of the training images and testing images, all nine videos are used for training. The trained model in this paper will likely have relatively poor performance on data for other scenes. In future work, we will extend the database and test the model’s performance on different scenes.


The RE.S.I.STO project targets visitors of Pisa medieval city, with the goal of providing high-quality digital contents accessible with smart devices. We describe the design, implementation and the test phases of the RE.S.I.STO application, whose goal is to automatically detect the proximity between visitors and artworks. Proximity is detected with a set of algorithms based on the analysis of Bluetooth Low Energy beacons. We detail our experimental campaigns which reproduce several museum layouts of increasing complexity at two pilot sites, and we compute the performance of the implemented algorithms to detect the nearby artworks. In particular, we test our solution in a wide open space located in our research institute and by performing a real deployment at the Camposanto Monumentale located in Pisa . The obtained performance varies in the range of 40% to perfect accuracy, according to the complexity of the considered museum layouts. We also describe a set of stress and stability tests aimed at verifying the robustness of the application during the data collection process. Our results show that the mobile application is able to reduce the beacon loss rate, with an average value of 77% of collected beacons. With the growing number of IoT-ready devices and their rapid diffusion in our daily lives, in the last decades, we witnessed the pervasive adoption of mobile apps delivering location-based services . The posstandard de facto localization solution for indoor spaces (similarly to GPS), leading to a still active and attractive research question.Therefore, in the last few years, a growing interest has been observed in research topics addressing the estimation of the location of mobile users both indoor and outdoor. This goal still represents a challenging task for indoor scenarios, where solutions based on Global Navigation Satellite System (GNSS) are unfeasible. Estimating the location of a target requires the adoption of technologies and data analytic tools that most often cannot exploit satellites as walls, wireless signal interference and obstacles significantly reduce the strength of satellite signals in indoor settings. To overcome this issue, in the last years several methods have been proposed, exploiting different kinds of wireless signals and technologies. Furthermore, the design of indoor proximity/localization solutions strictly depends on the target service. For example, the accuracy required to identify the occupancy of a room might be far lower than that required to guide a robot along a path. As a matter of fact, there is no Indoor proximity detection systems are of remarkable importance since they allow to provide the users with space-bounded services in a wide number of scenarios, such as health sector, industrial safety, urban mobility, emergency management, contact tracing, etc. For what concerns cultural environments such as museums or exhibitions, proximity detection systems can provide the users with context-aware information based on their closeness to points of interest thus improving both their experience and entertainment. To be easily adoptable in this kind of environments, proximity systems should be based on widely diffused technologies, be easy to use for the visitors, require a straightforward setup phase, provide high level of effectiveness and the user devices should be able to work for sustained periods of time on battery power. Despite that a multitude of techniques are available in the literature, they always entail a trade-off among those aspects. These challenging tasks keep this research field open to new approaches which can provide further analysis in real experimental scenarios.In this paper, we design and test a mobile cross-platform application, based on the Bluetooth Low Energy (BLE) technology, to automatically detect the proximity between visitors and points of interest, e.g., museum’s artworks. In turn, such application will be adopted to provide to visitors the correct digital content of the available artworks. Some remarkable works already addressed the problem of localizing people in a museum or to estimate their proximity with artworks, we refer to ,4,5 for The performance assessment of two proximity detection algorithms based on the RSS analysis of the collected beacons. More specifically, we compare the results in two pilot sites with different museum’s layouts of increasing complexity and we show how the performance varies when the target artwork is identified within the first, second and third option;The analysis of RSS’s fluctuations in which we study two key-metrics: the beacon loss rate measured with commercial BLE tags and smartphones and the RSS variation in indoor environments caused by the usage of three advertisement channels. Under this respect, it is worth noticing that we propose a possible strategy implemented in the RE.S.I.STO app to limit the side-effect of such frequency hopping based on the use of a specific quantile of the beacon’s RSS values.Our goal is to be able to correctly estimate the proximity with artworks without any prior knowledge about the followed path. To this purpose, we describe in this paper the design and test of the RE.S.I.STO app. We implement two different proximity detection algorithms and we compare their performance in terms of a classification problem in two pilot sites. More specifically, we measure how much our algorithms are able to correctly identify the nearby artworks against a detailed ground truth. Each artwork is equipped with a Bluetooth tag that emits BLE signals at periodic intervals with a pre-defined power of emission. Our tests also include a stress and a stability assessment of the performance of the app. In particular, we quantitatively measure the beacon loss rate at different conditions with the goal of measuring the amount of collected beacons with respect to the expected number of beacons. More specifically, we set up stress and the stability tests: stress tests measure the maximum number of collected beacons for a relatively short time period. Differently, stability tests aim at measuring the amount of collected beacons for a long period. We compute the beacon loss rate, as the ratio between collected and expected beacons averaged with respect to the different runs of each test typology. Our experimental results return an average of 77% of expected beacons with commercial smart devices and we reach an accuracy and F1 score varying between 100% and 40% according to the type of museum layout and of the pilot site. We report below the main contribution of this work:The paper is organized as follows: Digital guide applications are available for most museums worldwide. However, solutions designed to deliver recommended contents based on the user’s proximity with artworks are much less frequent . By explWe report in advertisement and the transmitted packets can be received from any BLE-enabled device, which are today extremely diffused. Several works are based on BLE beaconing such as 11], the Ultra-wideband (UWB) is a radio technology for transmitting very short pulsed messages that are spread over a large radio spectrum. UWB-based systems employ the trilateration approach which can estimate the position of a device within few centimeters of accuracy and do not require a line-of-sight path between the transmitter and the receiver, even though, in this latter case, the performance deteriorates. The works presented in ,17 exploNear Field Communication (NFC), based on RFID close-range contactless connection, is another technique used for proximity detection. These systems use the radio waves emitted by a reader (smartphone) to detect and query the data stored on RFID tags which are embedded in compact enclosures such as buttons or cards. Several works such as SMART VILLA , WolfsonRealistic scenario: we analyze if the solutions have been tested in a real museum or in a simplistic environment;Device heterogeneity: we report if authors tested their solution with a variety of devices or, differently, if only a specific device model has been used, e.g., ad hoc hardware. This aspect is crucial for the performance evaluation as BLE-based solutions might be affected by different BLE chipsets estimating the signal strength of BLE messages differently;Complex path: the complexity of the path followed to test the proximity with points of interests also affect the overall performance. We analyze if authors selected a trivial or nontrivial experimental path;Commercial devices: we further analyze the device adopted in order to report if commercial devices have been adopted;Robustness tests: we report if the proposed work also provides information about the robustness of the proposed solution. More specifically, we are interested in solutions working with long-lasting monitoring sessions, reproducing a realistic museum visit;Preliminary RSS data analytics: we analyze if the works provide a preliminary RSS analysis of the collected data so that to characterize the signal’s features of the BLE messages used to estimate the proximity. This analysis is important to better understand the indoor signal propagation for the considered testing scenario;Real-time outcome: we finally analyze if the solutions are designed for a real-time proximity estimation or for off-line evaluation. In the first case, the solution can notify to the user the proximity with a specific point of interest, while, in the second case, the proximity is detected off-line.We further analyze the BLE-based works we previously surveyed with the purpose of highlighting the distinguishing features of such systems running in a museum environment. To this purpose, we report in In , the autWith our proposed solution we try to address all these critical aspects and provide a robust and commercially available architecture ready to be deployed in a real world scenario.Vision-based techniques rely on the detection of both peculiar image features, such as points, edges or regions, or on visual two-dimensional pattern such as bar codes or QR codes . To acquQR or bar codes represent a cheap technology and the physical tag is able to store enough data, such as an ID or a website URL, to fetch the description of the artwork from the back-end server. However, in a museum environment, the usage of QR codes might not be a viable solution since they both interfere with the artistic visual and they require the user to take a photo, which is often prohibited in this kind of scenarios .The authors of  present As reported in We now describe the design of the RE.S.I.STO app with specific attention to the technology used and the proximity detection algorithms that we implemented and tested.the Beacon Logger Service (BL);the Proximity Detection Service (PD);the UI Content Viewer Service (UI).The RE.S.I.STO app implements a multi-platform mobile application based on the React Native Framework see . The appParsing the beacon payload;Extracting the information required for the PD service;Notifying the PD service with a set of the received beacons.The cooperation between the three services is implemented with a set of API. We report in BL interacts with the PD service by invoking the ProximityDetectionInterface. PD elaborates the received information, by running a suite of proximity detection algorithms . The DOM is the logical tree structure that describes all the elements in a page while the Virtual DOM is a layer of abstraction on top of the DOM which allows to translate abstract components to platform-specific native components during the rendering phase. The translation is performed by the so-called render bridge, which talks to the platform’s native API to generate the native components and render them on the screen. The bridge sits between two other main components: native modules and the Javascript virtual machine. These two components run in separate threads and they communicate through the bridge as shown in Concerning the performances, although Javascript is not as fast as native code, differences will not be noticeable in most cases. Moreover, the most demanding code could still be moved to a native module to remove the performance bottlenecks. On the negative side, React Native requires a not negligible effort to setup the initial development environment and might expose compatibility, dependency and debugging issues, especially, when dealing with some platform-specific modules. Overall, to develop cross-platform UI rich apps, it is among the most indicated platforms.In this subsection, we describe the algorithms we implemented to detect proximity between visitors and artworks. Such algorithms have been integrated with the RE.S.I.STO app. The objective of these algorithms is detecting the proximity between visitors and artworks in historical museums. More specifically, we do not track the position of the visitor in real-time, as generally done with indoor localization algorithms. This helps us to avoid one of the classical issues in this kind of algorithms such as a time-consuming calibration phase.Most of the commercial devices natively allow estimating the strength of Bluetooth signals emitted by tags, this measure is generally referred to as Received Signal Strength (RSS) and it is expressed in dBm units. Our algorithms rely on the RSS values collected from the app to obtain accurate proximity estimations. However, in order to mitigate the non-linearity between the RSS value and the distance of the emitting device, state estimation filtering techniques are applied, in order to provide better proximity estimation.N. Even if this is the most adopted filtering technique, it is not enough to stabilize the RSS values. To this purpose, we introduce the p-percentile filter  see , we compThe option: we compute the metrics by considering if the correct artwork is contained in the first option, in the first 2 options or if it is contained in the 3 options;The layout: we compute the metrics by considering 3 layouts of increasing complexity.Concerning the pilot 1 reported in Location-based services offer the possibility of delivering functionalities to end-users by inferring their location or the proximity with specific points of interest. In this paper, we present the design and a quantitative assessment of the reliability and performance of the RE.S.I.STO application, which is used to automatically detect the proximity between visitors and artworks. In particular, we present the RE.S.I.STO app, which is targeted to visitors of museums in the medieval city of Pisa . We first present the design of our cross-platform mobile application, based on the React-Native framework. We detail the architectural design and how we logically split the implemented software modules to collect sensing information and to elaborate on it for the purpose of detecting the artwork proximity. Our mobile application relies on the Bluetooth Low Energy protocol and, specifically, on the iBeacon technology. We implement two proximity algorithms and we detail our experiments composed of stress, stability and calibration tests. We show the performance of the app both in terms of beacon loss rate, accuracy and F1 scores. In particular, our stress and stability tests demonstrate the robustness of the app during the beacon collection task. We test the app with long-lasting session (up to 5 h of continuous monitoring) and we measure the ratio between expected and collected beacons. We then analyze the performance of the app during the proximity detection. This work also investigates some issues related to the calibration caused by the use of multiple advertising channels which imply a high variation of the RSS values at fixed distance. We also discuss how to mitigate the RSS fluctuations by computing a statistic of the received beacons’ RSS. To this purpose, we reproduce three typical museum layouts in a wide indoor environment and we compare the artwork ground truth with the outcome of the algorithms. Our results show that we can successfully detect the correct artwork with an accuracy up 95% for layout 1 and by considering the first option provided by the app. As well reported in


Robotic-assisted partial nephrectomy (RAPN) is a tissue-preserving approach to treating renal cancer, where ultrasound (US) imaging is used for intra-operative identification of tumour margins and localisation of blood vessels. With the da Vinci Surgical System , the US probe is inserted through an auxiliary access port, grasped by the robotic tool and moved over the surface of the kidney. Images from US probe are displayed separately to the surgical site video within the surgical console leaving the surgeon to interpret and co-registers information which is challenging and complicates the procedural workflow.We introduce a novel software architecture to support a hardware soft robotic rail designed to automate intra-operative US acquisition. As a preliminary step towards complete task automation, we automatically grasp the rail and position it on the tissue surface so that the surgeon is then able to manipulate manually the US probe along it.A preliminary clinical study, involving five surgeons, was carried out to evaluate the potential performance of the system. Results indicate that the proposed semi-autonomous approach reduced the time needed to complete a US scan compared to manual tele-operation.Procedural automation can be an important workflow enhancement functionality in future robotic surgery systems. We have shown a preliminary study on semi-autonomous US imaging, and this could support more efficient data acquisition. Developments in robot-assisted laparoscopic surgery have enabled highly dexterous instrument manipulation with enhanced ergonomics which facilitate precise movement within the anatomy without direct access to the surgical site. This has led to significant growth in robotic surgery as an alternative to traditional laparoscopic surgery . Yet desRobotic-assisted partial nephrectomy (RAPN) is one of the most common robotic-assisted surgical procedures . RemovinComputer-assisted interventions (CAIs) in RAPN have focused on enhancing surgical navigation and improving the management of US imaging probes or the fusion of endoscopic, US and CT modality data. This has been approached from both the image processing and understanding perspective where US information can be used to infer information such as tissue deformation  and fromIn this paper, we present a new framework for automated localisation and placement of a pneumatically attachable flexible rail (PAF) , 13 usinTrajectory generation using dynamic time warping motion planning;A control scheme based on visual servoing using endoscopic images during the pick-and-place of the rail;A comparison between two different surface registration techniques applied to ex vivo porcine kidneys ;Pre-clinical study of system performance with five surgeons comparing semi-autonomous and manual execution of the same task.Albeit a preliminary study, we investigate and compare the behaviour of expert surgeons and novices in their use of the device and their experience of the algorithm and workflow for automation.More specifically, the paper proposes a new platform architecture, algorithm and a pre-clinical user study with five surgeons. This has the following specific contributions:The dVRK system was used as the surgical robot underpinning our experiments. One of the patient side manipulators (PSM1) is equipped with a large needle driver (LND), while the other PSM2 holds the Pro-Grasp tool. Figure A custom ROS architecture was developed embedding different software algorithms, shown on the right part of Fig. /, e.g. /ws. A 3D point represented in Cartesian space is expressed through the vector of the components, e.g. Scalars are represented by plain letters, e.g. The fiducial localisation error (FLE) allows to measure spatial data points during image guidance  and in tlocalised points and the known checkerboard dimensions following co-registration. The acquisition of localised points will be explained in the experiments section. FLE can be mathematically formulated as follows:n represents the number of selected points, and localised and known points.The FLE is estimated by calculating the average of the measured distance values in terms of the Cartesian position between the The PAF rail has a fiducial represented by a checkerboard composed of squares with a side length of 1 mm used to estimate its pose. The pattern could be easily replaced with any other clinically compatible solution available in the market . The cheKnowing the rail’s geometrical model, it is possible to compute the location of the grasping site using perspective-n-point (PnP ) pose esr, which is the probed Cartesian position. The results coming from each method are compared in terms of distances between the real makers representing the ground truth.Registration of kidney soft tissue for image guidance is important since the kidney surface represents the target structure for positioning the image guidance rail. Our experimental set-up follows a previous registration comparison  for phanTo automatically place the rail on the target position with the robotic tool, two main tasks are needed: generate a trajectory to position the rail and develop a control strategy to optimise the operational performance.We separate the pick and place task in four stages which are shown in Fig. home position defined in the 3D space by STEP I, the tooltip starts from the grasping site in the central part of the rail. STEP II, the robotic tool approaches the home position.STEP III, the tip moves back to the predefined target point target point is computed as the centroid of the bounding box generated by the four ink markers.Finally, in STEP IV, the tool holding the rail moves towards the kidney surface to reach the W defined as distance matrix (DM). The DM has the dimension of the element of P in the equation. The values inside each cell of the DM are computed as: Dynamic time warping (DTW)  was usedgrasping position is estimated, it is then compared with the respective component of the kidney surface. Since the two objects are located on the same table surface, their depth estimation cannot differ more than the thickness of the kidney itself. This works as a safety initial control to ensure that the rail tracker is working correctly.During STEP II, it is important to guarantee the correct orientation of the tool in order to achieve a solid grasp. This can be done tuning the last three joints of the PSM arm that are indicated with A further control policy based on visual servoing was added to enhance the performance of manipulating the tool, starting from STEP II. The system does not present any tool tracking node, but once the relationship between the rail and the tool is geometrically established, after the grasping phase, it is possible to infer the same information. The position of the tooltip is extracted dynamically and transposed in the Transformations between images and the robot coordinates were computed and the accuracy of the tooltips’ position was examined through experiments. Forty-five image pairs of a 7 row by 10 column checkerboard acquired from different endoscopic poses were used as input for the stereo calibration  plot Fig.  was usedRegarding the kidney registration, the accuracy is quantified by the error in the markers reconstruction for each method in terms of Euclidean distance between the markers themselves. Given the ground truth of 50 mm, the value obtained from surface tracking with the PSM tip is 51.23 ± 0.44 mm, while with the ECM triangulation of surgical ink markers is 54.10 ± 0.88 mm. Unsurprisingly, given the small baseline 5.4 mm of the stereo camera in the da Vinci endoscope, localisation registration with PSMs was more accurate than with endoscope-based technique. These results do not affect the experiment negatively since probing techniques can be potentially computed in real surgical environments./ws. Although the starting location of the rail was always the same, due to errors coming mainly from the triangulation, the Cartesian position of the grasping point slightly varies among the 6 repetitions. The values related to the z-axis are greater than the other two axes, confirming that the main error component is due to depth estimation. Those values are still small enough to guarantee a correct grasp in all the repetitions since the length of the tool gripper jaws can compensate them. Based on these results, we decided to proceed on building the architecture on the visual feedback coming from the endoscope instead of using external cameras with increased baseline. This would improve the accuracy of the results but move us a step further away from an environment more similar to the surgical one. Fig. 42 seconds.Results from the repeatability test are shown in Fig S” followed by a number. Comparing the execution times between the automated task and the one executed by surgeons, it is possible to see how Table In this paper, we have reported preliminary experiments showing that automatic positioning of a PAF rail is possible by combining motion planning and visual servoing. Our framework can be used to place the PAF rail onto tissue by autonomous instrument motion following a planned trajectory and subsequently the rail can be used to manoeuvre a US probe. We implemented and compared calibration accuracy of this approach with two different dVRK instruments. Experimentally our pre-clinical case study showed surgeons’ interacting with automation of procedural sub-tasks. Results highlighted the need to build inherent user flexibility and make the system compatible with every tool, not only the LND and Pro-Grasp, meaning that design process for the rail system and handles can be improved. The proposed solution, although in a preliminary stage, showed promising results in terms of execution time. Multiple difficult challenges remain for translating such technology within more realistic experiments and a clinical environment. Examples include robust vision algorithm for taking into account tissue deformation or coping with dynamic effects like bleeding that obscures information inference for visual servoing. In addition to the positioning of the PAF rail, automation of the US probe manipulation requires additional motion planning and adaptation to tissue geometry. Further work is also needed for adaptive control in the presence of physiological motion and more comprehensive clinical workflow studies are necessary including the use of a functional US probe. These are some of the aspects ascribable to the failed executions highlighted in Fig.


Selecting a vaccine for fighting a pandemic is one of the serious issues in healthcare. Novel decision models for vaccine selection need to be developed. In this study, a novel vaccine selection decision-making model (VSDMM) was proposed and developed, based on the analytic hierarchy process (AHP) technique, which assesses many alternatives (vaccines) using multi-criteria to support decision making. To feed data to the VSDMM, six coronavirus disease-19 (COVID-19) vaccines were selected in a case study to highlight the applicability of the proposed model. Each vaccine was compared to the others with respect to six criteria and all criteria were compared to calculate the relative weights. The proposed criteria include (1) vaccine availability; (2) vaccine formula; (3) vaccine efficacy; (4) vaccine-related side effects; (5) cost savings, and (6) host-related factors. Using the selected criteria, experts responded to questions and currently available COVID-19 vaccines were ranked according to their weight in the model. A sensitivity analysis was introduced to assess the model robustness and the impacts of changing criteria weights on the results. The VSDMM is flexible in terms of its ability to accept more vaccine alternatives and/or more criteria. It could also be applied to other current or future pandemics/epidemics in the world. In conclusion, this is the first report to propose a VSDMM for selecting the most suitable vaccines in pandemic/epidemic situations or any other situations in which vaccine selection and usage may be deemed necessary.  Decision making in healthcare includes a complex set of pragmatic interactions among many stakeholders . LimitedA prioritization approach for health technology assessment was developed based on AHP and results highlighted the importance of treatment effectiveness, patient safety, and societal aspects in the decision-making process for best treatment alternatives in dialysis  and KoreAHP uses actual measures, such as price, counts, or subjective opinions, as inputs into mathematical matrices . The AHPImportantly, there is a shortage in the literature in factors and mechanisms for the selection of a vaccine among vaccine alternatives. Therefore, the development of a discrete choice model to investigate the different criteria for vaccine selection among available alternatives is critical. In this study, a vaccine selection decision-making model (VSDMM) was developed. Six criteria were identified to support vaccine selection: (1) vaccine availability which depends on the accessibility to the vaccine and its manufacturing location; (2) vaccine formula, which includes recombinant, subunit, inactivated, viral vector, DNA, and RNA; (3) vaccine efficacy , which cUsing this model, decision makers can select the most relevant vaccine for each situation. It is important to note that this is a proof-of-principal study that aims to highlight the applicability of this model to the decision-making process for COVID-19 vaccine selection.The proposed methodology included the following five phases: (1) identify criteria that control vaccine selection for a certain pandemic/epidemic; (2) propose and develop a VSDMM that can support vaccine selection from the available alternatives; (3) select the case study and alternative vaccines (six vaccines in the current case study); (4) collect data to feed and test the model, and (5) apply the VSDMM model on the case study and analyze the results. The proposed criteria included vaccine availability, vaccine formula, vaccine efficacy, vaccine-related side effects, cost savings, and host-related factors. We developed an AHP-based decision model that can provide support for the selection of the most suitable vaccine from many available vaccine alternatives for the COVID-19 pandemic. The problem was organized in hierarchical terms then the relative importance of each criterion was determined by pairwise comparison to other criteria. The first level of questions to decision makers included questions about the importance of a criterion as compared to another criterion regardless of the nature of the vaccine. The second level of questions to decision makers included questions about the importance of a particular criterion for one vaccine as compared to the other vaccine. The model comprised one matrix in its first level (6 × 6) and six matrices in the second level (6 × 6). Comparisons were performed by utilizing the preference scale .In this study, we proposed a hesitant fuzzy AHP method to help health policymakers evaluate the importance of intervention strategy alternatives for the COVID-19 pandemic . DevelopSelecting a vaccine is a multiple criteria decision-making (MCDM) problem with potentially competing criteria. The data for feeding and testing the model were collected through brainstorming sessions which is a typical documentation technique for collecting data. Two brainstorming sessions were conducted with two expert scientists in virology and vaccines. Respondent 1 (first session) was a professor of vaccine immunology and clinical microbiology, whereas respondent 2 (second session) was a professor of virology. The first step was to determine the importance and weight of each criterion. In the first level, the model was fed with values of the pairwise comparison matrices for the six identified criteria. Using the decision model, the relative weights of the criteria were calculated for the first and second sessions, as shown in Sensitivity analysis was conducted using the VSDMM for the data from respondent 1 as an example to showcase the effects of weight changes of the selected criteria on the model results. The weight of each criterion was increased or decreased by a value of 1 . The impact of this change was analyzed.After identifying the vaccine alternatives and their respective characteristics, the model was applied based on the selections of the respondents, which were based on their opinions and experience. The responses of the two selected experts that were fed into the model are shown in Based on the responses of the two experts, the relative weights for all vaccines were summarized in In addition, the first respondent preferred AstraZeneca (22.38%) followed by Janssen (21.56%) while the Sputnik V vaccine came in the last place (7.04%) . On the As shown in COVID-19 continues to spread, rapidly challenging governments and decision makers in different parts of the world . This glAmong the six vaccine selections included in this case study, mRNA-based vaccines (Pfizer-BioNTech and NIH-Moderna) and non-replicating viral vector-based vaccines (Oxford-AstraZeneca and Janssen) were considered . A healtIt is best for the decision makers to consult specialists in vaccines, in addition to specialists in other fields such as cost analysis and social aspects. The number of experts needed to make a good decision depends on the particulars of the model and other relevant conditions. Importantly, media coverage can certainly affect the people’s selection of vaccine but should not affect the conclusions of decision makers.The AHP technique has many advantages, such as flexibility and ease of use, through using different variables in selection and comparing alternatives based on multiple criteria. Moreover, the model is suitable for use in different countries with slight modifications that are based on selection criteria and available alternatives. On the other hand, most of the mathematical models have limitations. Two limitations appeared during the application of the proposed model in the study. The first arose when the consistency ratio (CR) of the results exceeded 10%, which could negatively impact the accuracy. This issue can sometimes lead the model user to ask respondents to reconsider some of their selections, which may not always be acceptable to the respondents. Another limitation is that there can sometimes be a need to merge some of the criteria that are close to each other, which cannot be simply achieved by decreasing the number of alternatives.In summary, the evaluation and selection of suitable vaccines for epidemics/pandemics can be complex given that each vaccine selection has advantages and disadvantages. In addition, the criteria used to compare vaccines are controlled by different factors. An AHP-based model for decision-making was developed to support vaccine selection and was applied on a COVID-19 vaccine case study (model) as an example that can be expanded upon for the current pandemic and future pandemics. Results obtained from sensitivity analysis indicated that the ranking order of alternatives remained consistent, even when the weights of the criteria were slightly changed. Decision makers can always modify or add to the comparison criteria or to the number of vaccine alternatives. The decision can also be made based on inputs from numerous decision makers (and not just two). These advantages can give decision makers the ability to select the best vaccine that suits their specific situation using the VSDMM.