Vehicles today have many advanced driver assistance control systems that improve vehicle safety and comfort. With the development of more sophisticated vehicle electronic control and autonomous driving technology, the need and effort to estimate road surface conditions is increasing. In this paper, a real-time road surface classification algorithm, based on a deep neural network, is developed using a database collected through an intelligent tire sensor system with a three-axis accelerometer installed inside the tire. Two representative types of network, fully connected neural network (FCNN) and convolutional neural network (CNN), are learned with each of the three-axis acceleration sensor signals, and their performances were compared to obtain an optimal learning network result. The learning results show that the road surface type can be classified in real-time with sufficient accuracy when the longitudinal and vertical axis acceleration signals are trained with the CNN. In order to improve classification accuracy, a CNN with multiple input that can simultaneously learn 2-axis or 3-axis acceleration signals is suggested. In addition, by analyzing how the accuracy of the network is affected by number of classes and length of input data, which is related to delay of classification, the appropriate network can be selected according to the application. The proposed real-time road surface classification algorithm is expected to be utilized with various vehicle electronic control systems and makes a contribution to improving vehicle performance. The recent advancement of vehicle control technology has resulted in several different driving assistance systems that offer convenience and safety to drivers ,2,3. HowRoad surface condition information can be used to improve driver safety and comfort in a variety of ways ,6,7,8. RMany previous studies have been conducted to estimate road surface conditions or the road friction coefficient. The methods used in these studies can be divided into three categories: (1) tire force-slip-based methods, (2) road property-based methods, and (3) tire–road interaction-based methods.The tire force-slip-based method has been applied most often in previous studies. Based on a vehicle and tire dynamics model, the behavior of the vehicle and tires are measured, the tire contact force and slip ratio (the speed ratio between the driving and driven wheels) are estimated, and the resulting force-slip relationship is used to determine the potential grip of the vehicle on the road ,11,12,13A road property-based method classifies the road surface by extracting the physical properties of the road surface by using noncontact sensors, such as cameras, light detection and ranging (LiDAR), radar, ultrasonic sensors, or optical sensors ,16,17,18In tire–road interaction-based methods, road surface conditions are classified by measuring tire vibration or deformation caused by the friction force on the tire contact patch ,21,22,23Among the various types of sensors that can measure tire–road interactions, intelligent tire sensor that directly measures the acceleration or deformation of tire elements through sensors installed inside the tire have been applied in many recent studies ,26,27,28To measure the response to tire–road surface interactions during the contact period under various road conditions, an intelligent tire system with a three-axis accelerometer capable of measuring up to ±500 g was installed inside the tire, as shown in To collect the vehicle driving status information, the steering angle, wheel speed, and brake on/off signals were acquired from the vehicle internal controller area network (CAN) at a sampling rate of 100 Hz in synchronization with the intelligent tire sensor signal. Since the vehicle internal CAN signal was used, additional sensors other than the intelligent tire sensor were not utilized. To receive signals from intelligent tire sensors and vehicle CAN network simultaneously with time synchronization, DAQ system composed of cRIO-9063 + NI-9853 of National Instrument was utilized.To collect an intelligent tire sensor signal database for deep learning, driving experiments were conducted on five different road surface conditions  at the proving ground (PG) of the Korea Intelligent Automotive Part Promotion Institute (KIAPI). For each road condition, the driving experiment was repeated at various vehicle speeds . The vehicle was driven in a straight line while keeping the vehicle speed as constant as possible without applying the brake. It was impossible to drive at a perfectly constant vehicle speed because a person gave the driving command, but the driving test was conducted with an error of less than 10 kph from the target vehicle speed. Due to operational limitations on the gravel and unpaved roads, the experiments on these surfaces were conducted at up to only 50 and 20 kph, respectively. For each road surface and vehicle speed, the experiment was repeated until sufficient experimental data with a total effective driving distance greater than 800 m, which corresponds to more than 400 instances of contact of the intelligent tire sensor with the road surface.n moving with constant intervals. The length of time window n is the same as the length of sequential data input to the network, and each data cut by the time window becomes one data for training, verifying or testing. Since the three-axis acceleration signal cropped by the time window has different magnitudes depending on the vehicle speed, the average and standard deviation are normalized to 0 and 1, respectively, so that the network excludes the effect by magnitude as much as possible and can learn only the waveform characteristics related to the road condition. Equation  and stacked on the training or testing dataset. Through this process, a training dataset with 100,000 samples  and a testing dataset with 20,000 samples (4000 samples per road surface) were acquired for each x-, y- and z-axis acceleration sensor signal. The training and testing datasets were acquired from independent effective learning data periods to prevent the network from learning the testing dataset in advance.Intelligent tire sensor datasets for each road surface type are trained with two types of DNNs: a fully connected neural network (FCNN) and a 1-dimensional convolutional neural network (CNN).An FCNN is a basic structure of an artificial neural network consisting of a series of fully connected layers that connect every node in each layer to every node in the next layer and finds a nonlinear relationship between input and output data by optimizing the internal parameters of dense networks . The majA CNN is specialized kind of DNN architecture that can be used to train spectral variations and model spectral correlations existing in the signals and is most commonly applied to learning images . TypicalFor two types of training networks (FCNN and CNN), the training was repeated by changing the structure and parameters of the network until the highest average accuracy from the training results with x-, y- and z-axis acceleration datasets (To find the optimal network with the highest accuracy, the same dataset was repeatedly trained by FCNNs and CNNs by changing the number of total layers, number of nodes in fully-connected layer, number of filter and filter size of convolution layer, and filter size of maxpooling layers. A network with a too-complex structure overfits the training dataset, reducing the accuracy of the test dataset, and a network with a too simple structure cannot acquire all the information in the training dataset, so it is important to find a network with an appropriate complexity.The final structure and parameters of each network are expressed in The accuracy of the network was calculated as the proportion of cases where predicted class by the network and actual class matched among the entire test dataset.The training results in In contrast, the training results in Examining the learning result of the CNN in more detail for each class reveals that the accuracy of the learned network varies depending on road surface class. For example, for the CNN training result based on the s result . As seenAlthough fully connected networks can be applied very widely due to their agnostic nature, FCNN tend to perform worse than special-purpose networks, such as CNN that extract patterns from sequential data in this case. Therefore, it is difficult to improve the accuracy of FCNN beyond a certain level, while trained CNN achieves higher accuracy by identifying important signal features that occur during the contact period for each type of road surface.Not only the type of network and dataset but also the normalization process of the training dataset greatly affects the performance of the overall learning result. If the CNN is trained using As a result of training the same CNN structure on four classes by combining dry asphalt and wet asphalt with 1mm waterfilm, which are the classes with many confusions at 5 class classification, trained network have accuracy of 95.7%, 86.1%, and 97.9% for As much as selecting a network with an appropriate structure, it is important to correctly determine the number of classes and the type of dataset according to the purpose of use. If the road classification result is used in the vehicle chassis stability control logic used in extreme conditions, such as heavy rain and off-road conditions, it is desirable to have high accuracy for roads under these conditions and it would be more proper to have a 4-class classification strategy, as shown in If a network is trained with all the axes signals from the intelligent tire sensor at the same time, the performance of the road condition classification network can be improved from the networks using only single axis signals. By copying the layer block from the first convolutional layer to the second pooling layer of Network in From a real-time application point of view, the proposed road condition classification algorithm has a practical advantage in that it enables estimating the road surface condition at any time instance regardless of the position of the wheel, while previous studies ,27 requiThe Finally, we analyzed the effect of the time window length on the performance of the CNN network. The learning results shown earlier were acquired using a time window length of 1000. Therefore, if this network were to be applied to estimate the condition of the road surface in real time, a 1 s delay would occur. Because a 1 s delay in road surface condition estimation would reduce the control or notification performance, the delay should be reduced as much as possible. Therefore, a CNN model with the same structure and parameters is trained while steadily reducing the time window size. As a result, it will be the most optimal network to adopt the shortest time window length that satisfies the minimum accuracy required for the network referring to In this study, a database containing tire–road interaction information in contact patches was collected on various road surfaces by using an intelligent tire system with an acceleration sensor attached to the inside of the tire. Based on this database, two types of deep learning algorithms are trained and their performance is compared to classify the road surface type. The results showed that a 1-dimensional CNN is more suitable than a FCNN for extracting road surface identification information from acceleration sensor signals. The results also indicate that x- and z-axis acceleration signals contain more information on the type of road surface than the y-axis acceleration signal, and the learning result based on the z-axis acceleration dataset has the highest overall accuracy. Additionally, by analyzing how the accuracy of the trained network is affected by the number of classes and the length of the time window, a criterion for selecting an appropriate network was suggested.Compared with the existing research on road surface classification, the proposed algorithm is more practical because it can provide road surface conditions continuously and fast enough to be utilized in the chassis controller of a vehicle. Moreover, due to the characteristics of the DNN, the developed road classification network can improve its performance by using an additional experimental dataset. For example, for a rapid acceleration period where the accuracy of road surface estimation is currently inferior, the performance can be improved by additional learning with the experimental datasets for the acceleration driving condition. In the same way, by securing additional datasets for various vehicle conditions such as tire type, tire pressure, vehicle load, and various driving conditions, the road classification network can be improved so that it can operate robustly in all situations.The currently developed algorithm classifies the road surface condition with respect to the acceleration sensor signal for only 1 axis but future research will be conducted to improve the accuracy and usability of the road surface classification by using the 3-axis acceleration sensor signal at the same time or by using the driving condition, vehicle body acceleration and gyro signal.


The widespread outbreak of the novel coronavirus disease COVID-19 has posed an enormous threat to global public health. A different set of policy interventions has been implemented to mitigate the spread in most countries. While the use of personal protective equipment and social distancing has been specifically emphasized, South Korea has deployed massive testing and contact-tracing program from the early stage of the outbreak. This study aims at investigating the effectiveness of testing and contact-tracing to counter the spread of infectious diseases. Based on the SEICR (susceptible-exposed-infectious-confirmed-recovered) model, an agent-based simulation model is developed to represent the behavior of disease spreading with the consideration of testing and contact-tracing in place. Designed experiments are conducted to verify the effects of testing and contact tracing on the peak number of infections. It has been observed that testing combined with contact tracing may lower the peak infections to a great extent, and it can thus be avoided for the hospital bed capacity to be overwhelmed by infected patients. It is implied that an adequate capability of testing and contact-tracing may enable us to become better prepared for an impending risk of infectious diseases. Since the first reported outbreak in December 2019, the novel coronavirus disease COVID-19 has swept all over the world and posed a significant threat to global public health. As of April 2021, more than 130 million confirmed cases have been reported, which accounts for about 1.7% of the world population [South Korea, witnessing the first local outbreak of COVID-19 in late January 2020, has adopted the intervention measures of massive testing and contact tracing to control the epidemic country-wide from the early stage of the outbreak. The idea behind testing and contact tracing is that the infected individuals are tested as early as possible and then isolated from others to prevent further disease transmission. Recognizing that the early detection of infections through tests is imperative to counter the highly infectious disease, South Korea has implemented innovative, large-scale testing facilities, including drive-through screening stations and temporary screening office and worked closely with the private sector to secure an adequate supply of tests from the onset of the pandemic . FollowiTraditionally, mathematical models are widely used to examine the transmission behavior of an infectious disease. Most of the mathematical models are compartmental ones, with the population divided into classes and assumptions being made about the rate of transition from one class to another . The susWith the capability to mimic complex systems involving uncertainty and explicitly addressing interactions among individuals in the system, the agent-based modeling (ABM) approach may seem a promising alternative. ABM is a computational model representing complex systems composed of autonomous individual agents or entities that can interact with each other under the same system environment ,17. It hWhen an individual has a symptom and gets tested, it takes a certain amount of time (defined as test turn-in time) to have the test results back. The individuals who take the screening test may move around until they get the test results, which may transmit the virus to others. When an individual is confirmed positive, two lines of intervention activities take place: case isolation and contact tracing. The COVID-19 positive individuals are hospitalized or isolated from others so that they do not spread the virus any further. Along with the case isolation, contact chains of the test-positive individuals are traced, and those who have been in contact with the infected individual are tested and self-quarantined for a certain amount of time. If an individual identified by the contact tracing process is tested positive, a new contact-tracing process is again initiated. The importance of contact tracing is highlighted by its impact on the transmission by asymptomatic patients. Different from the symptomatic patients who show various symptoms, such as fever and coughing, the asymptomatic patients do not exhibit symptoms at any time during the course of infection. Studies suggest that people infected with COVID-19 can transmit the virus whether they have symptoms or not. Asymptomatic individuals are not likely to get tested because they have no symptoms, so that they infect others while they are infectious. Transmission through asymptomatic patients could only be prevented by tracing contacts of the confirmed case and testing those contacts. For better effectiveness of contact tracing, the reduction of delay between symptom onset and case confirmation (case isolation) is critical because the infectious patients can infect others without knowing their own infection during this time. The isolation delay is composed of two factors, testing delay and test turn-in time. To the best of the authors’ knowledge, no existing studies explore the combined effect of testing and test turn-in time affecting isolation, contact tracing, and quarantine, which can play a pivotal role in mitigating the spread of COVID-19. This study presents an ABM simulation model to emulate the dynamic behavior of COVID-19. Non-pharmaceutical interventions such as testing, test result confirmation, contact tracing, isolation, and quarantine are explicitly included in the model at an individual level. The NetLogo 6.2.0 platform  is used s), infectious asymptomatic (Ia), infectious isolated with test positive (C), and removed (recovered or dead) (R). The state transition diagram is given in s + Ia + C + R. The state of individuals changes over time according to the transition probabilities. No births and deaths are assumed because the time horizon is relatively short compared to the human lifespan, apart from the death caused by COVID-19. A virus-infected person should be in one of the states, E, Is, Ia, and C. However, only individuals under states Is or Ia are infectious.The proposed model is based on the SEICR model with some modifications to include the unique characteristics of COVID-19. The population is divided into six different groups: susceptible (S), exposed (E), infectious symptomatic before test confirmation  (see  for ODEss) and asymptomatic (Ia). Studies suggest that asymptomatic individuals are also infectious, but less likely to transmit the virus than symptomatic individuals [The ABM simulation model is initially set up with a certain number of people in the susceptible state S and the remainder in the exposed state E. Individuals in state E may become infectious after some period  and move to an infectious state according to the progression rate. The virus may be transmitted to susceptible individuals from infectious ones with the transmission rate whenever they contact each other. The susceptible who are exposed to the virus then move from state S to state E. The infectious individuals are divided into two classes based on the onset of the symptoms: symptomatic , and they move around in a random way. In our model, contact is defined as the presence of individuals in the same square grid at the same time slot. The contact is implemented in NetLogo by using a link agent connecting infectious individuals and susceptible ones in the same grid. Later, the links may be used for the contact tracing process. Using a population-based survey result of epidemiologically relevant social contacts, it is indicated that the average number of contacts per day per person is 14–16, with significantly high dispersion ,45. The This study employs the conventional 3-level factorial design to examine the effectiveness of testing and contact tracing to counter the spread of COVID-19. Three factors are considered: testing coverage, turn-in time of test results, and contact-tracing ratio, denoted by Factors A, B, and C, respectively. The testing coverage indicates that only a certain percentage of infected individuals may take the test, which clearly affects the spread. It is reported that 40–60% of infected people have mild or no symptoms and may pass the virus to others without being tested ,47. The ® Core™ i7-9700 CPU @ 3.00GHz. It takes 12.5 s on average to a single simulation run. The full factorial design with three factors with three levels each yields 27 treatment combinations, and five replications have been made for each treatment combination resulting in 135 simulation runs. Five responses have been recorded for each run; duration of the outbreak (Y1), number of deaths (Y2), cumulative number of infections (Y3), peak number of infections (Y4), and time of peak infections (Y5). The experimental design and average responses at each design point are presented in Simulation runs are performed on a desktop computer with an IntelThe simulated data have been statistically analyzed with response surface methodology (RSM) using MINITAB Release 19. Investigated are the main effects of individual factors and their interaction effects on the peak number of infections, which is one of the most important aspects for the proper management of hospital bed capacity to deal with infected patients. The functional relationship between factors and responses may also be estimated in a polynomial equation with RSM. The peak number of infections can be estimated by the polynomial equation, of which coefficients are summarized in Useful predictions can also be made within the region of experimentation. The peak number of infections with respect to testing coverage and contact-tracing ratio is predicted and depicted in a posteriori estimates of them may be partially obtained if available. Sensitivity analysis can be conducted to better understand the disease spreading with respect to these parameters. It is intuitive that the basic reproduction number As noted earlier, the spreading behaviors of infectious diseases are highly dependent upon epidemiological parameters. It is elusive and probably quite rare, however, to attain enough information on these parameters, and only A wide variety of policy interventions to cope with a serious and imminent threat from COVID-19 has been adopted by most countries. While working at an unprecedented pace to develop pharmaceutical measures of vaccines and therapeutics to stop the pandemic, protective measures such as the use of PPE and social distancing have been strongly emphasized to slow the disease spread in the meantime. South Korea has supplemented the policy interventions with massive testing and contact tracing from the early stage of the local outbreak. Whereas aforementioned protective measures have been investigated by a few previous studies, the effectiveness of testing and contract tracing has hardly been dealt with. An agent-based modeling approach has been proposed to examine the effects of testing and contact-tracing program on the disease spread. The simulation model is constructed on the basis of the SEICR model. A set of designed experiments is conducted with three main factors, testing coverage, turn-in time of test results, and contact-tracing ratio. Since closely related to the management of hospital bed capacity, the peak number of infections for various factor settings is analyzed using response surface methodology. As expected, testing and contact tracing may be highly effective for curve-flattening. It is also worth noting that interaction between testing and contact-tracing ratio exhibits a significant effect, implying that expanding testing capability alone may not be as effective without extensive contact-tracing and vice versa.The main contribution of this study may be described in two folds. First, the proposed simulation model may provide useful insight into the spreading behavior of infectious diseases under various circumstances. Furthermore, examining the impacts of different mitigation measures can be helpful when planning and implementing public health policies to cope with potential outbreaks. To the best of the authors’ knowledge, for example, the effects of turn-in time of test results have not been investigated in previous studies. It is suggested that faster turn-in can be as effective as contact tracing to slow the spread. Therefore, additional policy efforts may be placed on enhancing the community’s testing capability instead of focusing on potentially controversial contact tracing. Second, designed experiments are quite popular to investigate causal relationships in many application areas. This study proposes employing the principles of experimental design combined with agent-based modeling in epidemiological studies. The outbreak of an infectious disease is a highly complicated phenomenon, and it can be challenging to sufficiently explain the dynamics of disease spreading. With the increase in computing power, an agent-based modeling approach has drawn increasing attention for modeling complex systems, and it can be efficiently employed to better describe the behavior of disease spreading. It is expected that more research efforts should be placed on developing flexible and versatile epidemic models to better represent the reality of policy interventions


In this paper, we proposed a novel operation to reconstruction tetrahedrons within a certain region, which we call MFRC (Multi-face reconstruction). During the existing tetrahedral mesh improvement methods, the flip operation is one of the very important components. However, due to the limited area affected by the flip, the improvement of the mesh quality by the flip operation is also very limited. The proposed MFRC algorithm solves this problem. MFRC can reconstruct the local mesh in a larger range and can find the optimal tetrahedron division in the target area within acceptable time complexity. Therefore, based on the MFRC algorithm, we combined other operations including smoothing, edge removal, face removal, and vertex insertion/deletion to develop an effective mesh quality improvement method. Numerical experiments of dozens of meshes show that the algorithm can effectively improve the low-quality elements in the tetrahedral mesh, and can effectively reduce the running time, which has important significance for the quality improvement of large-scale mesh. However, mesh generation is only the beginning, the quality of the generated mesh often determines the accuracy of the finite element calculation. In many cases, the quality of a mesh cannot meet the requirements. Therefore, it is indispensable to improve the quality of the mesh through mesh improvement to meet the requirements. The quality of the mesh depends on the worst element. If there are too many low-quality elements in the mesh, the solution error will increase, and the solution time will become longer or even cannot be calculated8. For example, in the virtual surgery system, cutting is the basic operation. Virtual cutting of the soft tissue model changes the topology of the mesh model. In the process, the mesh model needs to be refined. Therefore, soft tissue cutting will produce some low-quality tetrahedrons13, which can result in deformation errors or reduced computational efficiency. Thus, mesh quality improvement plays an important role in soft tissue cutting simulation.The finite element method has a wide range of applications in scientific computing and engineering problems. When solving problems with the finite element method, the solution domain needs to be decomposed into a set of finite simple elements. The rapid development of mesh generation technology provides a strong guarantee for the application of finite element methods for numerical simulationCommonly used operations in existing mesh improvement methods include: (1) mesh smoothing. The smoothing operation improves the mesh quality by adjusting the vertex positions; (2) topology operation. The topological operations improve the mesh quality by changing the connections of vertices in the target area; (3) vertex insertion/deletion. Vertex insertion/deletion improves mesh quality by inserting new vertices into the mesh or deleting existing vertices in the mesh.14, including 2-3, 3-2, 4-4flip  how to obtain the optimal connection method of the target area; (2) how to complete the operation within an acceptable running time. The most commonly used topological operation is the mesh flip how to o20. Although the target has not been changed, that is, seeking the best tetrahedralization of a polyhedron with a certain number of vertices and faces, we provide some novel ideas: When selecting the excavation face, if there is a face with a dihedral angle greater than Merge interior vertices and eliminate the worst interior elements.Construct a memo for the quality and validity information of the known tetrahedral division.The first and third items can ensure that the algorithm runs in an acceptable time, and the second item can more effectively eliminate the worst element and improve the overall quality.The main contribution of this research is to propose a topology operation algorithm for local mesh reconstruction, MFRC (Multi-face reconstruction), and develop a tetrahedral mesh improvement method based on MFRC combined with smoothing operations. Although multi-face removal improves the mesh quality more effectively than basic flip, in our test data, the multi-face removal operation can only handle more than ten tetrahedral elements at a time, which is more than basic flipping, but still limited. MFRC solves this problem. The MFRC algorithm is inspired by SPR 25. Common topology operations include 2-2 flip, 2-3 flip (3-2 flip), 4-4 flip. As shown in Fig.  14.Topology operations improve the shape of tetrahedral elements by changing the connection between vertices15, which removes a single edge from mesh, along with all the tetrahedrons that contain it. Edge removal can be regarded as n-(2n-4) flip, whereas 3-2 flip and 4-4 flip are both simple forms of edge removal. As we can see in the middle of Fig.  16. Multi-face removal can be regarded as 2n-(n + 2) flip, 2-3 flip, and 4-4 flip are both simple forms of multi-face removal. Shewchuk proposed an optimization algorithm that can find the optimal edges and faces when performing edge removal and multi-face removal, respectively, so that the edge removal and multi-face removal operations can obtain the best results26. The algorithm uses a dynamic programming algorithm to find an optimal edge removal operation. Chen developed a new mesh local reconnection technique. This technique recursively calls a new method called shell transformation to optimize the tetrahedral mesh. The shell transformation can be regarded as an enhanced version of edge removal19. And thread-parallel can be combined with face and edge swapping and vertex insertion using in mesh improvement27. Multi-face retriangulation is illustrated in Fig. 17.Edge removal is proposed by Briere20. The input of the SPR algorithm is a polyhedral cavity containing low-quality elements. First, an initial face is selected, and then the optimal tetrahedral division of the polyhedron is found through exhaustive enumeration. Unlike the flip operation, the SPR algorithm can process a polyhedral cavity composed of 20–40 tetrahedral elements. Therefore, the SPR algorithm can obtain better results. Although the SPR algorithm can raise the local mesh quality to a higher level, it is difficult to apply SPR to specific mesh improvement methods. The reason for this situation is that the optimal tetrahedral division of polyhedrons is an NP-hard problem, and it is quite time-consuming to solve it through exhaustive methods. Although some optimization strategies are proposed to improve the performance of SPR28, the time performance in the actual application is still not completely satisfactory.Different from the above-mentioned flip operation, the SPR algorithm proposed by Liu et al. aims to improve the mesh quality by finding the optimal tetrahedral division of the specified local mesh18. They improved the worst tetrahedral element in the mesh by inserting a vertex inside or on the boundary of the tetrahedral element. Edge contraction is a common operation that can simplify the mesh efficiently32. Klingner et al. used edge contraction to control the element size in tetrahedral mesh18. Wicke et al. proposed a dynamic mesh improvement scheme, which is aimed at tetrahedrons whose quality is below the minimum threshold. In this method, edge contraction is used to delete tetrahedral elements whose quality is degraded due to the existence of too short edges33. If vertices insertion and vertices removal are used to improve mesh quality, they can be effectively combined with other operations36.Klingner et al. applied vertices insertion in mesh quality improvement which achieves an excellent improvement result but is time-consuming37. Lei et al. proposed a GPU-accelerated parallel algorithm for Laplacian smoothing in three dimensions38. However, Laplacian smoothing cannot guarantee whether the mesh quality can be improved, especially in the case of three-dimensional mesh.Smoothing can also be understood as geometric optimization, moves the vertices without changing the connection of vertices in the mesh. The most commonly used geometric optimization method is Laplacian smoothing48. Du uses a method based on CVT  for mesh smoothing50, Chen uses a method based on ODT  for mesh smoothing51, and Zhong uses a particle-based method for mesh smoothing52. All these three methods  define the global mesh energy and improve the mesh quality by minimizing the energy, so the worst element cannot be penalized, some local optimization methods (such as sliver perturber53 and sliver exuder54) can be combined to get better results. Freitag and Ollivier combined swapping and smoothing to improve tetrahedral mesh quality. They smoothed the inner vertices using a non-smooth optimization algorithm and obtained excellent results22. The boundary mesh has a large impact on the quality of the overall mesh, however, flipping operations have a limited improvement on boundary elements, and moving the position of boundary vertices might change the shape of the mesh model. Klingner used the non-smooth optimization algorithm for the inner vertices and implemented constrained smoothing for boundary vertices18. Franco et al. improved the tetrahedral mesh using moving mesh smoothing and dealt with curved boundary vertices using radial basis functions (RBFs)55.The more effective method is the optimized-based mesh quality improvement methodThe MFRC algorithm takes a cavity as an input which is formed by a group of tetrahedral elements containing the target element . We choose a face as the starting point of the algorithm and consider all the vertices that may form a tetrahedron with the initial face. If the quality of the formed tetrahedron is lower than the worst quality of the original local mesh or the formed tetrahedron is invalid, the next face will be found and processed. Otherwise, if the found vertex is an interior vertex, insert a new element into the cavity and update the boundary of the cavity; If the vertex is on the boundary, insert a new element into the cavity, divide the original cavity into at most three new cavities through the boundary of the new tetrahedron, and then process the new cavities separately. Process each face recursively until the cavity is filled. After completing the entire search process, the optimal tetrahedral division of the cavity can be obtained.Figure In the MFRC algorithm, the cavity that wraps the ill-shaped element must first be constructed, which can be a convex polyhedron or concave polyhedron. First, select the target low-quality element, then search for the neighboring of the target elements, and finally wrap the target element with the cavity.According to the position of the low-quality element in the cavity, there are four situations: (1) One vertex is on the boundary of the cavity, and three vertices are inside the cavity. (2) Two are on the boundary of the cavity, and two vertices are inside the cavity. (3) Three vertices are on the boundary of the cavity, and one vertex is inside the cavity. (4) All four vertices are inside the cavity. According to the cavity construction principle of the MFRC algorithm, the distribution of the vertices in the cavity directly affects the results of the tetrahedral division of the cavity. The processing of interior vertices will be introduced in the next section. The number of vertices in the cavity directly affects the efficiency of the MFRC algorithm. Therefore, in actual application, to balance the mesh quality and execution efficiency, the MFRC algorithm will strictly control the number of vertices in the constructed cavity.One of the keys to ensuring good element quality is to prevent unnecessary short sides. To eliminate the short edges or narrow elements in the cavity as much as possible, the vertices inside the cavity are merged in the MFRC algorithm.n interior vertices. The average length of the boundary edges is n interior vertices can form E, if When there are more than two interior vertices, the MFRC algorithm judges whether to merge them based on the distribution of the interior vertices. Since interior vertices will form new tetrahedrons with the boundary face of the cavity, the MFRC algorithm considers the relationship between the edges formed by the interior vertices and the boundary edges of the cavity to avoid generating tetrahedrons with a short edge as much as possible. Suppose cavity When there are 3 vertices inside the cavity, consider the inner angle of the triangle built with the three vertices. If the maximum inner angle of the triangle is greater than Since the cavity is constructed based on MFRC, the interior vertices that need to be processed are the vertices of the target low-quality elements. After processing, the low-quality elements in the cavity will be eliminated, and the short edge can be avoided to a certain extent.When selecting the initial excavation face, in addition to the worst quality face, we also consider the concave face. If there is a face F on the boundary of the cavity, and the dihedral angle formed by F and the adjacent face is greater than In the MFRC algorithm, if the candidate vertex is the inner vertex of the cavity, first insert a new tetrahedron, and then use the faces of the new tetrahedron to update the boundary of the cavity; If the candidate vertex is located on the boundary of the cavity, insert a new tetrahedron first, then divide the original cavity into three sub-cavities at most, and use the new tetrahedron as the boundary. By solving the optimal tetrahedral division of the sub-cavity, the optimal tetrahedral division of the original cavity can be obtained. When the concave face is selected as the initial face, the original cavity can usually be directly decomposed into sub-cavities.Choosing a concave face as the initial face does not mean that quality checks, validity checks, and intersection checks will not be performed when inserting a new tetrahedron, but that fewer checks will be performed. The vertex-face that has been checked will be saved in the memo so that it can be used directly in future checks.After selecting the initial face, it is necessary to select vertices that can form a tetrahedron with the initial face, and the search order of candidate vertices affects the convergence speed of the MFRC algorithm. The main consideration principles for candidate vertices sorting include (1) Consider whether the formed tetrahedron is valid, including checking the orientation of the tetrahedron and whether it intersects with other elements. (2) Consider the quality of the tetrahedron. Choosing a better tetrahedron is the original intention of the algorithm. (3) Consider the position of candidate vertices, adjacent boundary vertices > interior vertices > other vertices.In the process of finding the optimal tetrahedral division of the cavity, the quality information and effectiveness information of many tetrahedrons will be repeatedly calculated. Similarly, the tetrahedral division of some sub cavities will be calculated repeatedly. These repeated calculations will cause significant time consumption.The MFRC algorithm constructs a tetrahedron-quality table. When it is necessary to calculate the quality of a tetrahedron, first check whether the quality of the tetrahedron exists in the table. If it does not exist, add the quality of the tetrahedron to this table after the calculation is completed. In the MFRC algorithm, some sub-cavities generated by the initial concave face needn’t be recalculated and will also be saved in the memo. Similarly, the MFRC algorithm constructs a digging-face-candidate-vertex table to detect whether a pair of digging-face and candidate-vertex constitute a valid tetrahedron.To ensure the efficiency and convergence of the algorithm, we introduce the termination conditions of the MFRC algorithm. If the sequence is empty or when the maximum number of iterations is reached or the mesh quality cannot be further improved, the algorithm terminates. The time consumption of the MFRC algorithm is determined by the number of boundaries of the cavity of the target low-quality tetrahedral element. To ensure the efficiency of the algorithm and the success rate of the algorithm, we tested multiple sets of cavities, including convex cavities and concave cavities. And when in use, we can set the maximum number of elements contained in the cavity. Algorithm 1 illustrates the implementation process of the MFRC algorithm. In order to prove the effectiveness of the MFRC algorithm, we tested the performance of the MFRC algorithm when dealing with local mesh. The meshes used for testing are derived from the actual mesh model and extracted from the mesh improvement process. And it is the result of the MFRC algorithm after the topology optimization fails . Figure 28 uses the V represents the volume of the tetrahedron, and 28 is a special convex structure; all nodes in the polyhedron are co-spherical, and a total of 19 test meshes are constructed. The first example has 14 nodes and 24 faces. The other examples were created from the first example by inserting points by points. The last example has 32 nodes and 60 faces.Reference28, although we give the quality and the number of elements, we only consider the number of iterations. The test results are shown in the table below. From the data in Table In this article, we constructed two sets of data. The first one is to take points on the sphere relatively uniformly, generate a surface grid based on these points, and then insert point by point. The second one randomly picks points on the sphere, generates a surface grid based on these points, and then inserts point by point. However, since there is no mention of the way or position of node insertion in referenceThe purpose of this research is to develop an efficient tetrahedral mesh improvement method. To obtain better improvement results, the proposed method combines mesh smoothing, edge removal, multi-face removal and vertex insertion/deletion. The volumelength measure is numerically and can punish all types of low-quality tetrahedrons, so the volume–length measure is used in this study. Since the overall quality of the mesh depends on the quality of the worst element, the mesh improvement operation is only performed on the low-quality elements, with clear goals and saving time. The boundary elements in the mesh have a great influence on the quality of the mesh, but in many practical applications, the improved mesh needs to be consistent with the original geometric model. Therefore, in this method, it is not allowed to change the boundary elements. The entire mesh improvement process adopts the hill climbing. When performing each local improvement, the optimized mesh quality must be higher than the initial mesh quality, otherwise no operation is allowed. This method is based on the MFRC algorithm and combined with the following operations, including:Mesh smoothing In this method, Laplacian smoothing, and optimization-based smoothing are combined. First, perform Laplacian smoothing on the target mesh, if successful, continue smoothing, otherwise perform optimization-based smoothing. In each smoothing process, each non-smooth node is processed only once25. If the mesh quality  cannot be further improved, the smoothing process ends.Edge removal, multi-face removal For edge removal, the dynamic programming algorithm proposed by Klincsek52 is used to solve the optimal triangulation problem of polygons, so as to realize the optimal edge removal operation. For multi-face removal, the algorithm proposed by Shewchuk29 is used to find the optimal face that allows to be deleted.Vertex insertion, vertex deletion For vertex insertion, for a tetrahedron whose edge length ratio is too large, insert a vertex on the longest edge to split the tetrahedron. If the quality of the newly generated tetrahedrons is better than that of the original tetrahedrons, the insertion vertex and the new tetrahedrons are retained, otherwise the vertex is deleted and the tetrahedron is not allowed to split. For vertex deletion, on the one hand, it is used to process the interior vertices of the MFRC algorithm, and on the other hand, it tries to improve the mesh quality through edge contraction operations.53 with vertex insertion and vertex deletion. After the size control, we perform smoothing to overall mesh, using Laplacian smoothing combined with optimization-based smoothing. Then the main loop is executed, and the low-quality elements list is updated first at the beginning of each loop. Perform smoothing on the elements in the list. If smoothing fails, perform the topology process. If neither smoothing nor topology optimization can improve the quality of the target element, the MFRC algorithm is called. Note that after each local reconstruction , the local mesh is smoothed. When none of the above operations can improve the mesh or reach the iteration threshold or the mesh upgrade to the target quality, the iteration improvement ends. Algorithm 2 presented the whole mesh improvement method. In order to eliminate too long or too short edges in low-quality elements, we perform size control on the mesh8. (2) volume–length ratio. Dannelongue et al. proposed to use volume and the average edge length to characterize a tetrahedron56. V.N. et al. extended the work of Dannelongue and presented the volume–length measure. Volume–length measure can be calculated as q is the quality value, V is the volume of a tetrahedron and 57.To improve the quality of the tetrahedral mesh, the quality measure of the tetrahedral mesh should be determined. The following are some commonly used quality metrics of tetrahedron meshes: (1) dihedral angle. In the finite element method, large dihedral angles hurt the accuracy of simulation and small dihedral angles affect stiffness matrix conditioning58, Stellar18 and mesh optimization methods provided in the Computational Geometry Algorithm Library (CGAL). CGAL provides four mesh optimization methods, including two global methods: Lloyd smoother49 and ODT smoother51 and two local methods: Vertex Perturb59 and Sliver Exuder54. The global method first defines the energy function of the mesh quality, and minimizes the energy function by relocating all vertices in the mesh, thereby achieving the improvement of the mesh quality. The local method gives priority to the worst-quality element, the so-called sliver. Vertex perturb starts from the worst tetrahedron, tries to add a small disturbance to each vertex of the target tetrahedron and updates the connectivity of the vertices to eliminate sliver. Sliver exuder converts triangulation into weighted triangulation and eliminates sliver by adjusting the distribution of vertex weights. The numerical tests were conducted on a PC , Gmsh (http://www.gmsh.info/60) and OriginPro Learning Edition (https://www.originlab.com/OriginProLearning.aspx) are used for mesh rendering and scientific drawing. As far as we know, Stellar is the best open-source mesh improvement method. The purpose of Stellar is to optimize the worst tetrahedral element as much as possible, and efficiency is a secondary consideration. Mmg is an open-source software for remeshing 2D and 3D surfaces and volumes, and it is still actively updated. Moreover, Mmg provides an effective tetrahedral mesh improvement function. Stellar + MFRC means the combination of the Stellar method and the MFRC algorithm, that is, using the MFRC algorithm replaces the topology processing part of the Stellar method.In this section, we test our proposed mesh improvement method on some mesh models to verify the effectiveness and compare the results with MmgIn order to compare the effects of the MFRC algorithm and other methods, we compared the dihedral angle, volume–length ratio, distribution range of the dihedral angle and running time of different methods. Examples of multiple mesh models used in this article are shown in Fig. Consider the dihedral angle and the volume–length ratio of the mesh model. Tables Consider mesh improvement processing time. The processing time of mesh improvement depends on the mesh scale on the one hand, and on the initial quality of the mesh on the other hand. According to Tables In addition to the comparison of single methods, the paper also compares the combined methods. Combine Lloyd, ODT, and Stellar with Exude, Perturb, and MFRC respectively. It can be seen from the data in Tables Through comprehensive consideration of the worst quality of the mesh, the distribution of mesh quality, and the running time of mesh improvement, it can be concluded that:20. Taking into account the efficiency issue, the validity and optimality of the elements are used to prune, and the memory search is used to speed up the algorithm. On the other hand, when there are vertices inside the cavity, the interior vertices are directly repositioned or the vertices are merged first and then re-adjusted the position of vertices. When local operations such as edge removal and multi-face removal fail, the adjustment of interior vertices by the MFRC can eliminate the worst element, and the reconstruction of the cavity can further improve the quality of the local mesh.The MFRC method proposed in this paper is a local mesh optimization method. The difference between MFRC and vertex perturb and sliver exuder is that the MFRC is aimed at a cavity containing the worst element. The purpose is to find a better quality triangulation, and it does not strictly require the cavity to have Delaunay characteristic. The reason for ensuring the advantages of MFRC is that, on the one hand, the tetrahedron is directly constructed on the basis of the existing cavity. In the actual operation, the cavity often contains dozens of elements, and the operation in a larger range leads to better local solution, thereby improving the mesh quality to a higher levelThe mesh improvement method based on the MFRC algorithm can not only improve the quality of the input mesh to a higher level in a short running time . Under the default command, Mmg will insert a large number of nodes into the input mesh, causing the number of elements to become huge. The Stellar method can undoubtedly effectively improve the quality of the input mesh, but the running time limits the application of Stellar, and Stellar changes the number of elements more than the MFRC method. Lloyd and ODT are global methods, and better results can be obtained when combined with local optimization methods. In this case, the MFRC algorithm still got better results. Therefore, after considering the cost and benefit, for single method, the MFRC method is better than the tetrahedral mesh optimization method of Mmg, Stellar and methods provided by CGAL. Considering the combination of MFRC and other global methods, although the running time is slightly higher than the other combination methods, the lowest element quality and dihedral angle distribution both get better results. For the three combination methods of MFRC in the paper, MFRC+Stellar provides better optimization results, but considering the time consumption, in practical applications, MFRC+ODT or MFRC+Lloyd may be a better choice.This paper proposes a local mesh reconnection algorithm, MFRC, which can obtain the optimal tetrahedral division of the local mesh. MFRC considers the concave face when selecting the initial face, directly filters out the vertices that do not meet the requirements and builds a memo to record the calculated element quality information, effectiveness information and effective tetrahedral division. MFRC reasonably merges interior vertices to obtain higher-quality tetrahedral division. MFRC allows the mesh to be changed in a wider range, so when the routine topological operation fails, MFRC can often achieve better results. The experimental results prove that the MFRC method can effectively improve mesh quality and achieve a good balance between mesh quality and running time.Many mesh generation methods will create the worst element on the boundary. Due to the importance of boundary geometric features, the quality optimization of boundary elements is a very challenging research topic. Therefore, many researchers have conducted extensive and in-depth research on this, such as inserting vertices in boundary surfaces, constrained smoothing, and some research on feature preserving. Although the MFRC method does not focus on the boundary elements, it also improves the quality of the boundary elements to a certain extent. In the future, we will apply the MFRC method to the virtual surgery training system, and the geometric characteristics of the tissue  play an important role in disease diagnosis and treatment. Therefore, in the current research, in order to retain the complete geometric features of the model, we did not consider complex processing of the mesh surface. The optimization of surface elements will be part of our future research work.


The possibility of carrying out a meaningful forensic analysis on printed and scanned images plays a major role in many applications. First of all, printed documents are often associated with criminal activities, such as terrorist plans, child pornography, and even fake packages. Additionally, printing and scanning can be used to hide the traces of image manipulation or the synthetic nature of images, since the artifacts commonly found in manipulated and synthetic images are gone after the images are printed and scanned. A problem hindering research in this area is the lack of large scale reference datasets to be used for algorithm development and benchmarking. Motivated by this issue, we present a new dataset composed of a large number of synthetic and natural printed face images. To highlight the difficulties associated with the analysis of the images of the dataset, we carried out an extensive set of experiments comparing several printer attribution methods. We also verified that state-of-the-art methods to distinguish natural and synthetic face images fail when applied to print and scanned images. We envision that the availability of the new dataset and the preliminary experiments we carried out will motivate and facilitate further research in this area. The abundant availability of new technologies for generating physical documents such as printers and scanners has raised many concerns about their misuse, examples of which include generating illegal documents, misguiding investigations through the generation of fake evidence, or even hiding relevant evidence in criminal investigations. For instance, child pornography can be printed and distributed between pedophiles in order to avoid virtual monitoring from the police, and illegal amendments can be incorporated in printed contracts without previous notice. Furthermore, professional printers can be used to print fake currency and packages of fake products, causing several negative effects on the economy. Finally, printing and scanning can be used to hide the traces of image manipulation or the synthetic nature of the images, since the artifacts commonly found in manipulated, and synthetic images are not present or detectable after the images have been printed and scanned.As a countermeasure to the diffusion of counterfeited printed documents, most of the major manufacturers of color laser printers have signed a secret agreement with governments to let the printers include secret (invisible) yellow dots onto printed documents . Such doThe challenges posed by printed document forensics have pushed the multimedia forensics research community to look for viable solutions based on the analysis of the artifacts left by the printers on the printed documents. In general, printed document forensics can be split into three main research areas: (i) source linking ; (ii) detection of printed manipulated images; and (iii) detection of printed and scanned synthetic images. Solutions for printer attribution are mostly based on the analysis of the extrinsic artifacts contained in printed documents, with the most popular ones for laser printers being banding, jitter, and skewed jitters. The presence of these artifacts has been exploited by several works to identify the sources of printed texts ,20,21,22The research carried out so far notwithstanding, progresses in this area is hindered by the lack of large reference datasets. The few existing datasets, in fact, present at least one of the following issues: (i) they contain ad hoc data prepared for specific research only; (ii) the printed patterns are often simple ones, such as icons, text, and halftone patterns; (iii) most of them consider old and non-professional printers; (iv) they do not consider copies of the same printer brand and model; and (v) to the best of our knowledge, no dataset with complex fake printed images exists. This last issue is particularly important and challenging, as most of the artifacts used to detect image manipulations, such as the correlation between RGB channels, discrete cosine transform irregularities, and even illumination inconsistencies, are gone after the images are printed and scanned. This problem is worsened by the observation that printing and scanning back a manipulated image is one of the most powerful and simplest attacks an adversary can conceive of to fool manipulation detectors. The availability of a large reference dataset overcoming the above problems may be of great help to foster new advances in printed document investigations, concerning both the detection of manipulated and synthetic documents and the attribution of printed documents to the device that generated them.In this paper, we aim at filling said gaps by presenting a large scale dataset that can be used for both applications: source attribution and synthetic image detection. Due to their relevance in image forensics applications, the dataset focuses on face images. In particular, the initial version of the dataset (we are planning to update it continuously in the next years) is composed of images printed by several printers and scanned back with a high-quality scanner. The images in the dataset are divided into (i) pristine images, to be used for the source attribution problem; and (ii) synthetic face images generated by three different generative adversarial networks (GANs). The dataset is further split into several subsets, containing regions of interest with different sizes for the investigation of localized artifacts. To evaluate the difficulties associated with the forensic analysis of the images contained in the dataset, we carried out an extensive comparative study including several source attribution and synthetic image detection baseline methods.We present a large scale dataset of color-printed face images for digital image forensics purposes, such as source attribution and synthetic images detection (deep fake images).We increased the diversity of the images in our dataset to make it suitable for approaches working on images of different sizes. Full scanned images and regions of interest with different sizes are available.To the best of our knowledge, our dataset is the first large scale dataset with printed and scanned artificial images created with GANs such as StyleGAN2 , ProgresWe present the results of an in-depth comparative study conducted on the new dataset regarding several baseline approaches, including both data-driven methods and methods based on handcrafted features. The comparison regards both source attribution and synthetic image detection.In summary, the contributions of this paper are:The rest of this paper is organized as follows: in Several works have investigated the exploitation of the artifacts left by the printers into the printed documents to identify their source. Here we focus our works aiming at source linking after document scanning, as they are usually cheaper, non-destructive, and fast.Common surveys in the literature ,45,46,47Generally speaking, there are two kinds of clues in printed documents that could guide a forensic investigation aimed at identifying the specific source of the document: intrinsic and extrinsic signatures. Intrinsic signatures are introduced by the printing process itself, whereas extrinsic signatures are intentionally inserted into the printed material. Three of the most investigated intrinsic signatures in laser printers are banding, jitter, and skewed jitters. Eid et al.  characteWith regard to the attribution of printed texts , most of the techniques based on intrinsic signatures treat such a problem as a texture identification problem, as artifacts such as banding are not easy to be obtained from text . For thiA second set of techniques to identify the source of any printed document focuses on the intrinsic signatures of documents containing colored pictures. In this case, banding artifacts are more evident as more patterns are printed (including the background). In this regard, one of the pioneer works is the one from Ali et al. , where tAnother set of research on colored documents source linking treated intrinsic artifacts as noise. Choi et al.  discrimiFinally, a number of techniques aim at identifying the sources of printed documents regardless of their content. Ferreira et al.  proposedThe abundant development of source linking approaches for printed documents notwithstanding, the identification of image forgeries and synthetic images from a printed and scanned version of a digital image has received considerably less attention. One of the few works in this area has been published in , where sThere is a need for a publicly available dataset that grows through time to include modern printers with different technologies and manufacturing procedures. We expect that different printers manufactured at different times generate different artifacts in printed documents that cannot be detected by previous works.There is a need for multimedia forensic techniques able to detect deepfake printed images. This is a very challenging problem, since several artifacts, such as the correlation between RGB channels, discrete cosine transform irregularities, and even illumination inconsistencies in the digital image versions are usually removed by the print and scan process. In this way, although several adversarial attacks have been discussed in the literature , the priTherefore, although printed document forensics  has received much attention in the last few years, there are still several issues to be tackled before solutions applicable to real-world scenarios are developed. Among them, the following two issues are relevant for the present paper:Therefore, the present work aims at moving a first step towards the solution of the above problems. This is done by presenting a long term dataset addressing both tasks: real-world source attribution with modern printers, and deepfake detection in printed and scanned documents. The details of the dataset we have constructed are described in the following sections.In this work, we present a new dataset trying to minimize some of the issues of existing datasets. The new dataset, which we call VIPPrint (the dataset is named after the VIPP group), consists of two sections. The first one focuses on printer source attribution and solves some common limitations in previous works, such as (i) lack of diversity and (ii) lack of redundancy. Concerning the lack of diversity, the dataset contains printers of different models and printing resolutions. This is an important issue when considering source attribution in real-world applications such as anti-counterfeiting detection, where the printing resolution used for printing a counterfeited document or package is unknown. The inclusion in the dataset of diverse printers marks a significant difference concerning existing datasets, which usually look at artifacts associated with specific printing technologies at fixed resolutions. As to lack of redundancy, very few works have analyzed the effect of the presence of two or more printers of the same model and brand in the dataset, thus neglecting the overlapping effect associated with the presence of two identical printers. The second section of the dataset considers an important, yet understudied, problem in digital image forensics: the detection of synthetic fake images such as those created by Generative Adversarial Networks after print and scan.The importance of the new dataset for digital image forensics is twofold: (i) it can foster the development of novel solutions for digital image forensics capable of withstanding a print and scan procedure, and (ii) it can inspire new techniques for source attribution of fake colored documents printed by modern printers, thereby linking the fake content to the owner, or user, of the printer.Concerning the content of the images composing the dataset, we decided to consider face images. The first reason for such a choice is that face images are particularly relevant in many applications related to biometric recognition, criminal investigations, and misinformation. A second reason is the availability of large scale datasets of face images that can be used as a starting point for the construction of the printed and scanned dataset. Giving researchers the possibility to work both with the digital images and their printed and scanned versions can represent an added value in many applications. Finally, AI-based techniques to generate synthetic images are particularly advanced in the case of face images, whose quality has reached unprecedented levels with no or very few semantic artifacts the forensic analysis can rely on .The details of the two sections the VIPPrint dataset consists of are discussed in the following.To select the images to print in the first dataset, we choose images from a dataset that has particular importance in the digital image forensics literature. These images come from the original subset of human faces from the Flickr-Faces-HQ (FFHQ) dataset . We use We choose to print the images in the dataset with printers that are diverse enough to make the source attribution problem challenging enough for state-of-the-art techniques. The initial version  of such sub-dataset for source attribution contains 1600 printings from the printers listed in The printers used were available in normal conditions (that means they were not exclusively used to print our dataset). Most of the printers were realtively new (they were from weeks to years old), and some of them needed toner replacements while printing. As for the scanner, we used the scanner from the Kyocera TaskAlfa3551ci multifunctional printer  extracted from the full images set. The importance of the ROI sub-dataset for classification algorithms is three-fold: (i) it may filter only areas that are useful for recognition ; (ii) such areas can be input to techniques that require lots of data such as data-driven approaches; and (iii) they allow the classification of documents through the fusion of their ROIs classification, providing the most accurate results. Such a strategy was validated several times before in the digital forensics domain, such as in works for camera source attribution ,57, antiE of the image patches using the horizontal (H), vertical(V), and diagonal (D) sub-bands of the discrete wavelet transform as follows:N is the number of values in the sub-bands of DWT and M is the fixed size of the squared patches. Afterwards, we ranked the image patches according to their E and selected the top 10 energy patches per image. The patches selected in this way compose the RoI subdataset. We chose to calculate the Energy after the binary image is created as we are looking for areas with more edges, instead of those with the highest edge strength. This approach is quite useful when printer noise is hidden in the background or flat areas. For this second set of images, we choose patch sizes of To extract the ROI patches, we used an approach inspired by the one adopted in  to tacklDetecting if an image is a deepfake, i.e., if it has been artificially generated by a GAN, is an increasingly trendy topic in multimedia forensics. In the context of a criminal investigation, for instance, assessing that an image has been taken by a digital camera rather than having been generated artificially can be of fundamental importance to assess the trustfulness of a proof. As another example, in a social media scenario, detecting synthetic images may be useful to understand that a misinformation campaign supported by fake media is ongoing.So far, research in this area has focused on digital documents, as they are intrinsically linked to fake news in social media. Several strategies have been proposed to deal with such a problem, including analysing the co-occurrence behavior of pixels in RGB channels , cross-s16,000 pristine and 16,000 fake images generated by StyleGAN2 .3500 pristine and 3500 fake images generated by ProgressiveGAN .500 pristine and 500 fake images generated by StarGAN .To promote further research on this topic, we built a second section of the VIPPrint dataset, containing a very large number of natural and GAN-generated face images. Specifically, we printed and scanned a total of 40,000 face images using a Kyocera TaskAlfa3551ci . This new subset contained, for the StyleGAN2 case 1109.822 patches for the size In this section, we discuss the experimental setup we used to assess the difficulties associated with source attribution (a multiclass classification problem) and GAN image detection (a binary classification problem) on the images of the VIPPrint dataset. Specifically, we present the metrics used for the experiments, the experimental methodology, the statistical tests we adopted (when applicable), and the baseline approaches we tested together with their implementation details.Even if authentication and source linking are different classification problems , the performance achieved by different methods on such tasks can be measured with similar metrics, by paying attention to interpret them properly according to the considered task. The set of metrics we have used is described in the following.For binary classification problems, the recall, also known as true positive rate, indicates the percentage of correctly classified positive samples and is calculated asFor the multiclass problem of source attribution, a similar metric can be used, with the difference of considering the recall for each class and calculating the weighted mean for all classes as a final metric.As a metric complementary to the Recall, we are interested in the classification precision, which is the fraction of correctly classified positives out of all the instances classified as such in a binary classification problem . That isFor the case of source attribution (a multiclass problem), we considered the precision in a way similar to what we did for the recall. That means, we calculate the precision for each class and then consider the final precision as being the weighted mean of precisions from all classes.F). It measures the harmonic mean of precision and recall and is calculated as follows for the binary classification case:The most important metric for both problems is f-measure  divided by the number of samples under investigation:In the multiclass problem, we repeated the steps done for other multiclass metrics: we calculated the accuracy per printer and then the weighted average of all the accuracies for all classes was calculated.To validate the experiments carried out on the VIPPrint dataset, we followed two different approaches, depending on which application we were considering. In the source attribution scenario, we chose the gorithms  and was gorithms ,34. AccoIt is important to notice that, in contrast to camera source attribution validation approaches commonly used in the literature ,57 that For the GAN-image detection task, we took a set of detectors and trained them on the original digital images, as done in the original papers, and assessed their performance on printed and scanned images. We focus on the detection of the StyleGAN2 images in the VIPPrint dataset, as they are by far the best quality GAN images in the dataset. The procedure we followed to evaluate the performances of the detectors was a simple one: we used 24,000 digital images for training and 6000 digital images for validation, and then we used 2000 printed and scanned images from our dataset for testing the detectors. All the sets were independent and stratified .n rows (where n is the number of tested approaches) and ten columns, which were the ten f-measures resulting from the 10 runs. The test aimed at verifying whether the distributions of all the sets of f-measures changed significantly. We used the Friedmann test [To verify that the source attribution results were statistically significant, we performed a series of two tests in the ann test  for thisFor the post test, which tested the statistical relevance of each pair of approaches, we considered the Student’s t-test . This teIn this section, we briefly describe the baseline methods considered in our tests.For this problem, we selected 12 approaches divided into three sets. In the first set, which we call image texture descriptors, we used a set of common descriptors that are mainly used for image characterization. For a source attribution task, such descriptors can be useful to differentiate printers’ banding artifacts efficiently if the analyzed patterns do not change much, and therefore, they normally exhibit good performance in some printer source attribution tasks ,20,22,34GH) divides the grayscale version of the analyzed image into a fixed number of blocks. Then, a histogram of gray intensities is calculated for each block and all the histograms together are used to generate a description vector.The gray histogram  (hereaftHOG) extracts the edges in the image by means of the Sobel kernel gradients; then it computes the gradients for all the orientations. Finally, a histogram of such orientations is fed into the input of a machine learning classifier.The histogram of oriented gradients  (hereaftEH) is similar to HOG. However, it calculates, for each block, the dominant edge orientation instead of all of them, and the descriptor is a histogram of these orientations.The edge histogram   from Ferreira et al. [The multidirectional version of the gray level co-occurrence matrix .The multidirectional and multiscale version of the same approach proposed in  (GLCM-MDCTGF-3X3).The convolutional texture gradient filter in a 3 window  (CTGF-3XDWT-STATS).The 39 statistical features from the diagonal discrete wavelet transform sub-band from Choi et al.  .The 16 and 19 layer versions of the VGG convolutional neural network  (VGG−16 RESNET-50 and RESNET−101).The 50 and 101 layer versions of the RESNET convolutional neural networks  The Densely connected networks  (DENSENEINCEPTION-V3);The third version of InceptionNet  (INCEPTIXCEPTION)The InceptionNet evolution considering fully separable filters  ;A CNN that acts on three co-occurence intra-channel matrices  (CONET);CROSSCONET).A CNN that acts on six co-occurence matrices considering both intra- and inter-channel co-occurrences   for the final classification stage. We chose a linear kernel support vector machines classifier, as it is well suitable to deal efficiently with high-dimensional features. We performed a grid-search approach to find the best parameters to train the classifiers for each of the 10 experiments. This was done by applying a five-fold cross-validation procedure to the training data only. The classifiers’ parameter C was varied in the set We start with the feature engineering approaches. For  website . AlthougVGG−16, VGG−19, RESNET-50, RESNET−101, INCEPTION_NET-V3, and DENSENET and XCEPTION_NET. The final classification result for an image was set to be the mode of the classifications obtained on single patches. This approach is commonly known as majority voting and was also validated in printed document forensics research [CONET and CROSS-CONET networks for GAN image detection, which acted on https://www.tensorflow.org/, accessed on 11 February 2021), and Keras  libraries.In contrast to the previous approaches, those based on convolutional neural networks were applied patch-wise, with patches of size research ,22,34. TWe fine-tuned the neural networks pre-trained on ImageNet with the input training data , by initializing the weights with Imagenet pre-trained weights. We tried other procedures, such as fine-tuning, only the tops of the networks  and freezing the other layers, but the results were not worthwhile.In the fine-tuning procedure, we cut off the tops of these networks, replacing them with a layer of 512 fully connected neurons, followed by a 50% dropout layer and a final layer with eight or two neurons, depending on the task.The networks were trained with the steepest gradient descent optimizer , with anWe set the maximum number of epochs for source attribution to 300 epochs. However, after 20 epochs we implemented an early stopping procedure if the validation loss did not improve. For deepfake detection, we chose 10 epochs and the early stopping condition was implemented after five epochs, as we were using much more data.We used data augmentation for the source attribution task by using the following image processing operations: rotation, zoom, width shifts, height shifts, shears, and horizontal flips. For GAN detection, since much more training data are available , we did not use any data augmentation.For a fair comparison of the data-driven approaches, we used the following common procedure to train the neural networks:https://tinyurl.com/vipprint, accessed on 11 February 2021.Finally, all the data presented in this paper, including the two datasets, the scripts for generating the high-energy blocks, In this section, we discuss the results of our comparative study for both source attribution and GAN-image detection.An overall view of the average results we got for the 12 baseline source attribution techniques we have tested is reported in GH descriptor, for example, tries to discriminate printers by assuming that different printers print the same images using different colors, which is supposed to be seen in different histograms plotted in the n-dimensional space and clustered by hyperplanes such as those from the SVM classifiers. That assumption failsm as the resulting f-measure (0.53) is pretty similar to a random guess. The approaches relying more on the effects of gradients and edges (EH and HOG), where the banding and other printing artifacts are more evident [LBP descriptor (F = 0.75). A possible explanation for the better performance of LBP compared to other texture descriptors is that it explores gradient information by encoding, in several regions, the neighborhood relationships. This can better identify the behavior of printer patterns compared to other texture descriptors.The first aspect to be noticed in the results shown in  evident , achieveDWT-STATS (F = 0.76) dropped with respect to the performance reported in the original paper [CTGF-3X3 filters convolutional generated features, building their histogram in a gradient interval. Said approach yielded an average F = 0.79, which is considered a good result when compared with the common texture descriptors we considered and discussed in the previous paragraph. We can also see from GLCM-MD (F = 0.78) and GLCM-MD-MS (F = 0.84). These approaches consider more directions in the neighborhood of pixels and more statistics in the co-occurrence matrices. Additionally, for GLCM-MD-MS, more scales were used in order to achieve invariance with respect to the size of the printed pattern. Such features can be considered ad hoc texture features specific for laser printer attribution, thereby achieving better performance than general texture descriptors.The second set of techniques included approaches based on handcrafted features specifically tailored for the source attribution problem. To start with, we found that the performance of al paper , highligal paper  show betVGG−16 and VGG−19. They provide the two worst results for all metrics, while also showing a very high standard deviation, indicating very unstable training. Two possible reasons for such bad performance are the shallowness of the networks and their very simple architecture, including only convolutional and pooling layers. Those explanations were confirmed by the results gotten by deeper and more complex networks, RESNET-50 and RESNET−101 CNNs. These networks exhibited (by far) the top two results of our tests, with F = 0.91 for RESNET-50 and F = 0.90 for RESNET−101.Finally, the last set of techniques are based on CNNs . Let us RESNET-50 showed slightly better performance, as it is less affected by said problem and also because it classified 4 out of 10 printers perfectly, instead of 3 out of 10.To better investigate the differences between these networks, we start to discuss where they fail and succeed in the printer attribution task. As a final step of the printer attribution experiments, we analyzed the statistical significance of the results. By applying the Friedmann test to 12 vectors (one for each approach) with the 10 f-measures, we got a p-value lower than VGG−16 does not allow one to draw statistically significant conclusions for some of the comparisons, namely, those with GH, HOG, EH, and LBP. Other cases where no statistically significant conclusions can be drawn are the comparisons of DWT-STATS and LBP, and HOG with EH.The first noticeable behavior in RESNET-50  and RESNET−101 were confirmed by the results of the Student’s t-tests with all the other methods. At the same time, the difference between the performances of these two networks was not statistically significant. Based on these observations, we can conclude that RESNET-50 and RESNET−101 represent the better solutions for the source attribution problem, even though their best performance, with an f-measure equal to 0.91, along with the difficulties in distinguishing some Kyocera printers, leave room for further improvement.Finally, we notice that the superior performances of We now discuss the results of GAN image detection on printed and scanned documents. For that, we considered a set of CNNs trained on different patch sizes  they did not consider a dataset that grows with time, including more recent and professional printing devices; and (ii) they did not consider the authentication of printed artificial images.In this paper, we showed the extent of such limitations by validating existing authentication and source linking methodologies on a novel dataset specifically made for printed document forensics. The new dataset, the VIPPrint dataset, presents the first version of an ongoing effort to build a challenging environment for printed image forensics. To the best of our knowledge, the dataset contains the richest publicly available corpus of printed natural and artificial images, with 40,000 images addressing deepfake face-image detection, and 1600 images focusing on source attribution in a closed set of eight printer sources. The experiments we have run showed that this dataset results in an error probability of at least 9% for the best baseline source attribution methods. The dataset raises even more challenging problems in the case of GAN-image detection, given that StyleGAN2 images look like the original ones for all the tested methods after they are printed and scanned.The experiments we ran have guided us toward a lot of further work. First of all, we will continue updating the dataset to include new printers, more scanners, other GANs, and acquisition devices such as digital single lens reflex (DSLR) cameras. Second, we aim at investigating novel ways of selecting regions of interest in the digitized images and also considering other color spaces in addition to RGB. Finally, we are also headed toward investigating and applying adversarial attacks in the printed domain; we will add relevant features to our dataset in order to evaluate the effectiveness of printed document forensic methods.