This review focuses on disordered, or amorphous, porous heterogeneous catalysts, especially those in the forms of pellets and monoliths. It considers the structural characterisation and representation of the void space of these porous media. It discusses the latest developments in the determination of key void space descriptors, such as porosity, pore size, and tortuosity. In particular, it discusses the contributions that can be made by various imaging modalities in both direct and indirect characterisations and their limitations. The second part of the review considers the various types of representations of the void space of porous catalysts. It was found that these come in three main types, which are dependent on the level of idealisation of the representation and the final purpose of the model. It was found that the limitations on the resolution and field of view for direct imaging methods mean that hybrid methods, combined with indirect porosimetry methods that can bridge the many length scales of structural heterogeneity and provide more statistically representative parameters, deliver the best basis for model construction for understanding mass transport in highly heterogeneous media. The focus of this review will be on so-called disordered or amorphous heterogeneous catalysts see , rather The heterogeneous catalysts themselves come in a variety of types ,3. The aWhen the overall activity is diffusion-limited, then the performance of the catalyst depends strongly upon the pore structure ,10. UndeThe ‘anatomy’ of the title refers to the structure of the void space of heterogeneous catalytic materials and that of the complementary solid phase, where this is relevant to the nature of the void space over the many length scales seen in The pore structural characterisation techniques covered in this review include more recently introduced imaging methods. In the past, even though tomography theory was invented first , as showThe porosity, or voidage fraction , is the key characteristic parameter of a porous solid . HoweverIn addition, for many hierarchical porous solids with void features over many length scales, some of the voids may be below the imaging resolution limit. However, several methods were developed to map heterogeneous spatial distributions of porosity where the void space structure itself is below the imaging resolution limit and/or where the required field-of-view limits the resolution possible.For CXT, a high-electron-density contrast agent can be added to the void space to enhance the difference in X-ray absorbance between the void and solid phases. A variety of contrast agents were used, including tri-iodomethane , xenon gThree-dimensional ptychographic X-ray computed tomography (PXCT) is able to provide 3D maps of the electron density of the mesoporous solid matrix, and thence, can also map the spatial variation of the total (and not just surface accessible) mesoporosity . HoweverMagnetic resonance imaging (MRI) can also be used to provide maps of the macroscopic heterogeneities (>10 μm) in the spatial distribution of porosity ,35,36 s. PorositScanning electron microscopy (SEM) operating in backscattered electron (BSE) mode can be used to obtain local porosity measurements for the relevant field of view . This maWhile an imaging modality may potentially supply a complete direct visualisation of the void space of the porous solid, the simple visual examination of a 3D rendering of the structure on a computer may not be particularly informative. The form of the void space thus needs to be parameterised with suitable descriptors. In order to make the unwieldy whole void space more tractable for understanding the differences between materials, it is often partitioned into smaller fragments known as pores  proposedFor a 3D, regular lattice-based rendering of the void space obtained from imaging, a suitable image analysis algorithm can be used to identify the local hydraulic radius minima. One such algorithm is morphological thinning . This coIt is often suggested that the pore size distribution obtained from such a partitioning procedure from an image is likely to be more accurate than a (supposedly) more indirect method, such as gas adsorption. However, the physical processes involved in the multilayer build-up and condensation during gas adsorption are somewhat analogous to the abstract morphological thinning procedure. The adsorbate film tends to grow via the addition of successive layers as the gas pressure is increased, analogous to each generation of interfacial voxels added to the morphological thinning. Further, during adsorption, pore necks tend to fill with condensate before pore bodies, just as necks thin earlier during the application of the thinning algorithm compared with pore bodies. Hence, obtaining pore size distributions from gas adsorption is not so far removed from doing so via morphological thinning of an image of the same void space, as might be supposed. Indeed, recent simulations of gas adsorption using geometrical thinning algorithms were performed and found to be predictive of the form of adsorption isotherms for a range of more ordered and disordered porous solids ,44. FurtIt was mentioned above that the ratio of the pore size to the voxel resolution is the most important factor that determines the limit on the capabilities of a given imaging modality, especially for the ‘direct’ determination of pore sizes. In addition to the removal of image artefacts that might be falsely confused with pores, image segmentation is the key image analysis process that affects the accuracy of pore size measurements via direct imaging, as the method of deciding the boundaries of pores impacts their apparent size. This process remains prone to subjective elements in the various proposed algorithms and associated procedures .In addition, certain imaging modalities can be augmented to provide maps of the larger-scale spatial distribution of pore sizes below the voxel resolution. For example, NMR relaxation time preconditioning can be used to produce spatially resolved maps of the distribution of local average pore size over macroscopic length scales (>10 μm) using MRI ,36. ThesEven once individual pores are identified in some way, the characteristic size of pores in an amorphous material is ambiguous. Generalised definitions of characteristic sizes include the hydraulic mean diameter defined in terms of the surface-area-to-volume ratio of the pore . HoweverFor cylindrical pores, besides the diameter, another key parameter is the pore length, and thence, for the porous material, the pore length distribution. This is because refinements of the basic theory for the Knudsen diffusion mechanism showed that the Knudsen diffusivity depends on the pore length . For exaRigby and co-workers proposed a methodology to obtain pore length distributions that used the particular experimental dataset available from the fully integrated gas sorption and mercury porosimetry method  and analiS and pore volumes iV for each pore radius ir in the pore size distribution to obtain the exponent iα in a power law relation between the pore length and radius from a plot ofAn alternative method used to obtain pore length is that proposed by Pomonis and co-workers ,64. Thisiα, then it is subject to the usual uncertainties in the determination of the pore size distribution, namely, the mode of capillary condensation assumed and errors due to multilayer build-up in macropores if the capillary condensation does not pore fill the material at the top of the isotherm. This method was tested by comparing the pore anisotropy ib, which is defined as the ratio of the length to the diameter for both fresh  and functionalised templated silicas. This showed that, as might be expected, the addition of small functional groups led to an increase in b as the pore diameter was simply narrowed, while the addition of large groups led to a decrease in b as they blocked off pores and thereby shortened the remainder.If a pore size distribution is used to obtain the value of The pore length distribution can also be determined from image analysis of full 3D reconstructions of the void space, where the morphology was such that nodes (junctions) in the pore network can be clearly identified, and thus, the endpoints of individual pores are clearly demarcated. This is relatively straightforward where void spaces are reduced to the underlying graph of nodes and branches .The tortuosity of a porous catalyst is often used to obtain an effective diffusivity for coupled diffusion and reaction calculations. Effective diffusivity is typically defined as the product of the porosity and reference diffusivity, divided by the tortuosity factor. The nature of the reference diffusivity depends upon whether diffusion is in the molecular or Knudsen regime. Tortuosity is a void space descriptor related to its topology, but it is often conceptually vague and ill-defined ,67. ThisThere are also purely geometric definitions of tortuosity that are based upon deviations from the straight-line distance between two points in a void space. However, the particular choice of the pathway within the porous solid is not clear . The torIn any case, tortuosity, like porosity, is a parameter that can be defined for a particular volume of space, but, while not strictly a vector quantity, also potentially incorporates an implicit directional component that arises from the boundary conditions of the mass transfer process to which it refers. The type of mass transfer process studied in a measurement of tortuosity affects the value obtained ,34,35,36The more recent availability of imaging methods that can produce reconstructions of the void space has enabled the determination of geometric measures of tortuosity via image analysis algorithms . Fu et aTomographic and other 3D imaging methods only provide models of the void space. This is because so-called ‘direct’ imaging methods are not actually direct visualisations of the void space since, as described above, 3D tomographic images are only produced after the application of some sort of reconstruction algorithm to a series of 2D projections or similar . For exaOnce an image is acquired, it may contain artefacts and noise that obscure the details. Hence, subsequent image processing is a key step in deriving an accurate model of the void space ,79. Whil2g−1, respectively), until the accessible pore volume in the tomographic images was identified using impregnated silver crystallites as a tracer for the surface area to count towards the total. However, Yoshida et al. [The issues with the segmentation of the void space are exemplified by the recent attempt to compare the specific surface area obtained using a standard BET analysis of gas adsorption data and an analysis of electron tomography images for porous titania materials . Initiala et al.  did not More recently, it was suggested that higher-resolution electron tomography (ET) can be used to improve the segmentation of lower-resolution CXT images . Prior kAnother key issue with imaging approaches is achieving a sufficient sample of the void space so that it is properly representative of the whole . It is p3 μm at a resolution of 35 nm to characterise the macroporosity . These respective imaged volumes were used as the basis of a hierarchical model of the porous solid at the corresponding length scales. First, diffusion was simulated under a concentration gradient in the void space of the lower-length-scale model, while the solid was considered impermeable. This was converted into an effective diffusivity for use as the continuum transport parameter for the ‘solid’ phase in the larger-length-scale image model. Diffusion was then simulated in the larger-length-scale model with transport permitted through both the macroporosity and the permeable solid phase. The simulations were found to give rise to good predictions of the overall effective diffusivity for the alumina pellets measured using a Wicke–Kallenbach cell [Alternatively, the pore sizes obtained from 3D CXT, of a small elemental volume, can be compared with those from, say, mercury porosimetry data for many pellets from the same batch . If therach cell . Howeverach cell , with thHowever, while for some composite materials, such as some types of rocks, a finite  set of classes of mineral grain types can be identified and sampled via high-resolution imaging, for many industrial materials, such as catalyst pellets, no such discretised and finite classification of composite constituents can be achieved . This woThere are also certain pore-to-pore co-operative physical effects, such as advanced condensation  or advanced melting ,46,84,85Many studies of catalytic materials using images as the basis of the structural representation, sometimes combined with digital reconstruction, for predicting mass transport are, thus, only performed on various thin films , where the set of structural variability is similarly limited, as for specific mineral grains within rocks due to the relatively small size (compared with pellets) of the region of interest (~thicknesses of microns only) ,87,88. HFor reasons that are subsequently laid out below, it is asserted here from the start that all descriptions of the void space of disordered porous solids, even if obtained from supposedly direct imaging methods, are, in fact, only models of a more complex reality. This is because in all cases considered here, some details of the original material are left out of the description, and thus, it is necessarily a simplification, as the term ‘model’ suggests. This simplification may be due to the inherent limitations of the structural characterisation technique used and/or a purposeful idealisation that is intending a model that is not ‘maximally realistic’ . The ideGalilean idealisation introduces omissions from, or distortions of, the structure of the original material with the aim of achieving sufficient simplification that the description thereby obtained is then mathematically or computationally tractable, both for interrogating the description of the structure itself and using it to make predictions of processes occurring within . As mathThe level of simplification of disordered void spaces achieved with pore bond network models can be such that it is then possible to conduct a full simulation of complex, coupled mass transport and reaction processes for the pellet scale, including multiple modes of mass transport. These simulations can make a priori predictions of extrinsic reaction rates that can then be compared with an experiment to validate the model, though very few such studies have been performed ,104,105.The Galilean idealisation level of pore network models also permits entirely theoretical studies of how different arrangements of pore sizes can potentially impact catalyst activity and selectivity. Different void space geometries can be constructed on a computer, and then their performance for particular reactions can be tested ,108. TheMaps obtained from imaging of the macroscopic spatial variation in local average porosity and pore size of real materials can be used as the basis for the construction of heterogeneous but simplified, site- or bond-type, lattice-based models, where the image data supply the void space descriptors for the model units corresponding to individual voxels ,120. TheFurther, the simplification process can be reversed in so-called ‘de-idealisation’, wherein extra complexity is added back to a simplified model, such as the incorporation of  rough walls to previously smooth-walled, regular cylindrical pore bonds in network models . HoweverModels formed via minimalist idealisation are only of the degree of complexity necessary to include the particular causal factors that give rise to a phenomenon of interest that occurs within the porous solid . Hence, An example of a minimalist model is the so-called ‘brute force’ method for the prediction of transport properties of porous solids uses tomographic images as supposedly full, direct representations of the void space on which to perform mass transport calculations to predict transport parameters, such as diffusivity, tortuosity, and permeability. However, due to the fundamental limitations on the imaging resolution and field of view, even in the most favourable circumstances, the void space representation is still really merely a model. For example, foam-type materials are often used to make highly macroporous ceramic supports for extremely diffusion-limited and/or heat-transfer-limited catalytic systems  and are However, even the CXT-image-derived grid in Minimalist models are especially necessary for systems with high degrees of multiscale complexity since an exhaustive approach would be impossible with current computing capabilities. For some catalyst systems, multimodal imaging suggests that the pellet structure is highly heterogeneous over a broad range of length scales such that it would not be possible to comprehensively characterise enough of the pellet to be representative, and a hypothetical exhaustive dataset would be too large to manipulate (to predict diffusion and reaction) at present, even with current supercomputers ,126. In For example, multimodal imaging of methanol synthesis catalyst pellets involving CXT with different resolutions and FIB-SEM, as seen in Z and the overall lattice size L. The cylindrical pore bonds were characterised by individual pore diameters and a collective power law exponent for the general relationship between those diameters and pore length. These parameters were all obtained from a percolation analysis of the combined nitrogen gas adsorption and overcondensation desorption isotherm data for the pellets. Controlled modifications were made to the void space of the catalyst pellet via the entrapment of mercury in progressively smaller pore sizes using porosimetry, and then the model parameters were measured for the modified structure. As shown in τ for mass uptake into the modified pellet void space relative to that for the original empty pellet (τ0) is related to the change in the apparent size of the percolation model lattice size (L − L0) following mercury entrapment according tow is the pellet feed particle size; S/V is the external surface-area-to-volume ratio of the finished pellet; and the ratio of the geometry factors for the feed particles and whole pellet is denoted by γ, which is equal to unity if the feed particles and pellet have the same geometrical shape.However, despite the extreme levels of structural complexity in the pellets, as is evident in the images in Mousa et al.  proposedIf a random lattice as in a is usedThe CXT images of the invading mercury front in roll-compacted (RC) feed pellets shown in b demonstMultiple-model idealisation is ‘the practice of building multiple related but incompatible models, each of which makes distinct claims about the nature and causal structure giving rise to a phenomenon’ . This ap3 [6 such sample volumes, and thus, it is still prohibitive to fully image even just a single pellet, never mind a representative sample of several pellets from a heterogeneous batch. Notwithstanding this still-standing sampling issue, high-resolution imaging has enabled the more direct sizing of larger mesopores and the examination of pore connectivity directly, including the identification of relatively larger pore necks. This enabled the finding of unexpected results, such as that the pore size distribution curves remained of similar shape, average size and spread, up to quite long calcination times, and thus, high pellet densities [As mentioned above, increasing computer power has meant that ever-larger datasets can be manipulated, meaning that more extensive images or structural models can be developed. The imaging resolution with even lab-based, as well as synchrotron, CXT has steadily been improving. These ongoing developments in hardware technology are expected to continue. For example, studies were recently conducted on ~8 mm diameter porous alumina compacts with nanotomography using a phase-contrast holotomography setup on a synchrotron that was able to image with a pixel size of 25 nm for a field of view of 64 × 64 × 54 μm3 . Howeverensities .In order to study ever larger samples with tomography, the beam strength must be increased to enable it to penetrate the sample to produce the projections. However, the potential for increased beam-inflicted damage to samples still limits the beam strength that is possible with X-ray and electron tomography, and thus, the upper size limit of samples. However, there were recent developments in hardware, such as X-ray (and neutron) dark-field imaging using grating interferometry to allow for the quantification of sizes of features below the resolution . Dark-fiThere are also likely to be continued ongoing developments in mathematical techniques and software for data analysis. To overcome the trade-off between the field of view and resolution discussed above and to remove artefacts, artificial intelligence (AI) methods are increasingly being used for image processing ,131. DeeThe development of coupled characterisation and representational approaches for use with multiscale, hierarchical porous systems is also ongoing in order to cope with ever more complex materials. In the homogenisation approach, some averaging law is used to incorporate small-scale physical properties into an upscaled, continuum representation . This stThe structural characterisation of so-called amorphous or disordered porous solids still presents an outstanding challenge due to the often complex hierarchy of heterogeneities in void space structure in such materials. Key void space descriptors, such as porosity, pore size, and tortuosity, can now be obtained directly from imaging, but only for limited volumes of the void space. Other, more indirect, techniques allow for the mapping of these parameters for void spaces below the resolution of the image, but for more statistically representative sample volumes. Hence, the complex hierarchy of heterogeneities in the void space of disordered materials can be characterised sufficiently to construct meaningful models of many porous catalysts. The representations of disordered, heterogeneous catalysts are, broadly, of three basic types, which are dependent on the level and purpose of the idealisation of the model.It was seen that some catalyst pellets have a sufficiently homogeneous void space such that relatively small fields of view are representative of the larger whole, and thus, simulations of mass transport are predictive, along with more idealised random models. However, it was also seen that many other types of catalyst pellets are so heterogeneous that they have hierarchies of complexity over multiple length scales. While it was seen that multimodal imaging can detect this heterogeneity, a full (exhaustive) representation of the void space would be beyond the current computing power. However, it was also seen that more indirect but multiscale porosimetry methods, such as overcondensation and mercury porosimetry, can bridge the many levels in the hierarchy of complexity in one experiment, and can also sift out, or abstract, the key aspects of the void space controlling mass transport. In combination with complementary theoretical modelling approaches, such as percolation analysis, these methods can also extract an underlying commonality of basic form from an otherwise apparent diversity of complex patterns. For example, it was found that both the complex patterns  of mercury-filled surface clusters of large pores observed in SD and RC feed methanol synthesis catalyst pellets could be mapped onto the simpler surface clusters of random pore bond network models using gas overcondensation data. This type of idealised model captured the key aspect of the void space, namely, the surface clusters of larger pores that controlled mass transport in the modified pore structures, and thus, allowed the model to be predictive of the transport properties. In this case, the complementary imaging data helped to explain why the minimally idealised model was successful.Due to the multifarious types of void space in disordered porous materials, it is necessary to use some tool, such as the ‘sifting’ or ‘filtering’ approach described above , to dete


Laparoscopy is employed in conventional minimally invasive surgery to inspect internal cavities by viewing two-dimensional images on a monitor. This method has a limited field of view and provides insufficient information for surgeons, increasing surgical complexity. Utilizing simultaneous localization and mapping (SLAM) technology to reconstruct laparoscopic scenes can offer more comprehensive and intuitive visual feedback. Moreover, the precision of the reconstructed models is a crucial factor for further applications of surgical assistance systems. However, challenges such as data scarcity and scale uncertainty hinder effective assessment of the accuracy of endoscopic monocular SLAM reconstructions. Therefore, this paper proposes a technique that incorporates existing knowledge from calibration objects to supplement metric information and resolve scale ambiguity issues, and it quantifies the endoscopic reconstruction accuracy based on local alignment metrics. The experimental results demonstrate that the reconstructed models restore realistic scales and enable error analysis for laparoscopic SLAM reconstruction systems. This suggests that for the evaluation of monocular SLAM three-dimensional (3D) reconstruction accuracy in minimally invasive surgery scenarios, our proposed scheme for recovering scale factors is viable, and our evaluation outcomes can serve as criteria for measuring reconstruction precision. A precise three-dimensional reconstruction model that incorporates internal anatomical structures and laparoscopic positions can assist surgeons in making decisions and performing operations. Furthermore, the reconstruction of a surgical scene is an essential step for data registration in surgical navigation and surgery with augmented reality. As an approach to this, vision-based simultaneous localization and mapping (VSLAM) in miniature medical devices has gained interest . By applThe accuracy of three-dimensional reconstruction has a direct impact on the outcomes of subsequent research and applications. Evaluating the accuracy of three-dimensional reconstruction is not only for verifying the effectiveness and robustness of reconstruction algorithms and assessing the precision and completeness of reconstruction results, but is also more importantly for providing crucial references for further three-dimensional analysis and visualization. However, due to the specificity of application scenarios, it is challenging to acquire ground-truth data for three-dimensional endoscopy in minimally invasive surgery. Because different parts of endoscopy have varying degrees of complexity, as well as distinct features among the various organ tissues involved, it is hard to establish comprehensive three-dimensional endoscopic testing data; therefore, the main method still depends on other medical devices for measuring relevant morphological data . HoweverMoreover, the main disadvantage of monocular SLAM is that the absolute scale of the reconstructed scene and the camera movement are intrinsically unobservable. Without additional metric information, the 3D reconstruction results of a SLAM system are unclear. Typically, monocular SLAM methods estimate the absolute scale by acquiring extra metric information from distance measurements or an inertial measurement unit (IMU) . MethodsThe quality evaluation of the final modeling output of SLAM-based 3D reconstruction is typically divided into two categories: qualitative and quantitative evaluation. Qualitative evaluation involves a visual comparison, where human perception is used as a subjective indicator to assess the overall visual effect, as well as the details, edges, and other structural information within the reconstructed data. Quantitative evaluation, on the other hand, typically involves an objective assessment of the accuracy of the 3D reconstruction model based on errors and similarities in the data structure. To provide technical support for subsequent engineering applications and further research, quantitative evaluation of the accuracy of 3D reconstructions is particularly important in order to accurately reflect a system’s precision through corresponding error calculations. In special circumstances, such as when a patient’s objective physical conditions are poor and it is necessary to reduce the harm caused by ionizing radiation to a certain extent, external collection methods cannot meet the examination requirements of conventional CT or MRI equipment, resulting in the inability to use such equipment to collect three-dimensional data.In terms of quantitative evaluation for SLAM-based 3D reconstruction, it has been shown that high accuracy in camera trajectory estimation does not necessarily mean high-quality surface reconstruction. This means that trajectory errors cannot directly express the mapping accuracy of the system. Currently, the evaluation of 3D reconstruction results typically involves statistical analysis based on measured and reconstructed data, which means calculating the error between the reconstructed model and the ground truth model, followed by corresponding analysis and evaluation.After obtaining the three-dimensional ground truth of the target object, it can be used as a benchmark for quantitative evaluation of the accuracy of reconstructed 3D model. In terms of evaluating the accuracy of the point cloud model obtained from SLAM-based three-dimensional reconstruction, two main indicators are selected for statistical measurement.The overall absolute accuracy evaluation of the reconstructed point cloud model relies on obtaining high-precision point cloud ground truth using three-dimensional measurement instruments as a reference model. The point cloud obtained from SLAM-based three-dimensional reconstruction is transformed to the same reference coordinate system as the ground truth using manual or ICP algorithms. After alignment with the reference model, the error between the two models is calculated based on Euclidean distance, and the evaluation process flowchart is shown in This paper investigates and examines the prevalent methods for assessing the accuracy of SLAM three-dimensional reconstruction, and addresses the issue of evaluating the precision of cavity-reconstruction point cloud in real-world applications. Without adding extra equipment or sensors, this paper proposes two methods based on the ground control point (GCP) concept to employ scene calibration objects as prior knowledge to augment metric information and restore monocular SLAM scale factor, and quantify them with local alignment degree. The experimental results demonstrated that the reconstructed model restored the true scale information, revealed the overall accuracy of a laparoscopic SLAM system, and offered technical guidance for subsequent applications.To address the evaluation problem of 3D reconstruction in inner-cavity visual SLAM, using real data of human inner cavities has limitations and is difficult to fully test during the research stage. Therefore, based on the actual needs of inner-cavity scenes and limitations such as the equipment and data collection, this paper constructed a simulated inner-cavity experimental platform and used a simulation model to test the 3D reconstruction of inner cavities in a real indoor scene using an inner-cavity monocular SLAM system. In order to recover the scale factor of the inner-cavity monocular SLAM system and conduct accuracy evaluation, this paper combined two feasible methods: intra-cavity calibration object co-reconstruction and external pre-calibration, and mainly calculated the measurement error of the inner-cavity SLAM from the perspective of the absolute size error, and used the root-mean-square error (RMSE) to statistically calculate this error value. During surgery, inspection through laparoscopy focuses more on local measurement information of target areas or specific lesions. After obtaining the reconstructed point cloud model, typical areas of the target organ or soft tissue surface that are significant in the inner-cavity are selected as the target lesion area. Local conformity evaluation is adopted, and the quantitative evaluation of the system’s 3D reconstruction accuracy is carried out from the direction of size conformity of the feature structure.s is defined as follows:Before calculating the error between the reconstructed measurements and the real size, the idea of the GCP (ground control point) based on the significant ground control points is adopted, and the coordinate transformation relationship is corrected by establishing the coordinate transformation relationship in the base image data with precise information, so that the a priori knowledge obtained by other methods  can be used to establish the estimation relationship, and the point cloud model obtained from the reconstruction coordinates to the world coordinates with real scale. In the 3D reconstruction of the simulated internal cavity, the conversion relationship between the original coordinate system of the reconstructed model and the real-scale world coordinate system is obtained by the known conversion relationship between the actual dimensions of the calibrated object and its dimensions in the 3D point cloud model, so that the 3D reconstruction results can be corrected for the relative scale. Therefore, for the scale uncertainty problem between the reconstructed estimated size of the target object size in the environment and the actual true size, the scale scaling factor The intracavity calibrator reconstruction method uses the calibration template in the scene to align the 3D reconstruction results to the reconstructed coordinates with standard dimensions, i.e., the 3D coordinates of specific feature points in the reconstruction data under the real space are obtained, and then the dimensions of the target lesion in the internal cavity can be calculated, and the real structure dimensions measured in its reality are calculated and counted, and the dimensional fit of the reconstruction results of the internal cavity target can be obtained.In the 3D point cloud data-measurement analysis, PCL is used to extract the feature corner points of the target organ tissue surface in the map model using a manual point selection method for measurement. For example, the longest diameter of an irregular lesion can be obtained by using the lesion contour endpoints through the Euclidean distance calculation.N is the number of measurements.The evaluation index RMSE is calculated as shown in the following Equation (2), where A calibration template is placed in the scene and used as part of the environment for 3D reconstruction, as shown in To evaluate the 3D reconstruction of endoscopic-vision SLAM, using real data from human body cavities in the research stage has certain limitations and makes it difficult to conduct sufficient testing. Therefore, based on the actual needs and constraints of endoscopic scenarios, equipment, and data acquisition, this paper buildt a simulated endoscopic experimental platform, using a synthetic model to test in a realistic indoor environment, and performed 3D reconstruction of the simulated cavity using an endoscopic monocular SLAM system. In order to recover the scale factor of the endoscopic monocular SLAM system and evaluate its accuracy, this paper combined two feasible methods: co-reconstruction with an intracavitary calibration object and pre-calibration outside the body, based on the minimally invasive surgery scenario. Then, mainly from the perspective of absolute size error, we calculated the measurement error of endoscopic SLAM and use the root-mean-square error (RMSE) to quantify this error value. Moreover, considering that during surgery inspection using laparoscopy, more attention was paid to local measurement information of the target areas or specific lesions; we selected typical parts with prominent features on target organs or soft tissue surfaces in the cavity as lesion areas after obtaining the reconstructed point cloud model. We used a local alignment evaluation method to quantitatively evaluate the 3D reconstruction accuracy of the system from the direction of feature structure size alignment.This section verifies the effectiveness and reliability of the endoscopic monocular SLAM system in a simulated abdominal puncture exploration by building a simulated human body cavity experimental platform. It mainly focuses on conducting multiple endoscopic monocular SLAM reconstruction experiments around the designed intracavitary accuracy evaluation method and quantitatively evaluating the accuracy of the reconstruction data as well as analyzing and discussing the corresponding results. Thus, it studied the validity of the proposed evaluation method for this scenario and determined the precision of the system. In this section, the relevant experimental hardware configuration of the endoscopic monocular vision SLAM-based 3D reconstruction platform is shown in In the endoscopic 3D reconstruction experiment based on monocular SLAM, this paper simulated a realistic minimally invasive surgery scenario by using a synthetic abdominal cavity box and a straight rod laparoscope on a mechanical experimental platform, as shown in This paper uses Zhang’s calibration method to calibrate the intrinsic parameters of the laparoscope used in the experiment. The calibration work uses a 9 × 7 chessboard, with a single black and white square size of 24 mm. As shown in To enhance the calibration accuracy, we used the average pixel error of the images to filter and eliminate individual chessboard pictures, and finally obtain the remaining 15 pictures to calibrate again. After, the calibration was completed; the overall error of its results is shown in The corresponding internal reference data of the laparoscope can be obtained through laparoscopic calibration work, and the specific values are shown in Among them, Firstly, for the method of converting coordinate relations with the help of a known calibration template given in In the intracavitary reconstruction error evaluation experiment, our overall shooting condition was in a realistic indoor environment. We turned off all external light sources, and put the “stomach” organ model shown in The accuracy is calculated from the model by selecting the average measured distance from the source to the target points, and the experimental measurement comparison results are shown in The method of placing a calibration template in the scene must take into account the size or volume impact of the calibration template. Therefore, in special cases, some intracavitary spaces are too small and complex, and due to surgical operation or patient’s body reasons, it is impossible to introduce external tools or calibration objects into the cavity. In this case, the method of co-reconstruction using intracavitary calibration objects cannot be used. Based on this problem, this paper designed a method of pre-calibration outside the body, considering recovering the scale information of the monocular SLAM system in the cavity, by pre-calibrating through the laparoscope inserted outside the abdominal cavity. A calibration box for pre-calibration outside the body was made. During surgery, the laparoscope slowly passed through the calibration box before entering cavity. The geometric diagram is shown in K, The inner walls of the calibration box have two parallel planes on top and bottom. Therefore, based on the constraint condition of plane parallelism, by considering the pixel coordinate range and normal relationship of reconstructed map points, map points in the top and bottom plane regions were selected to obtain geometric expressions of two fitted planes. The geometric equations of the top and bottom planes were set as follows:s was obtained to recover the real scale of the monocular SLAM-based 3D reconstruction in the cavity.Point selection was repeated for fitting in order to obtain the best parameter estimation results after optimization iteration. Then, the reconstructed estimation value of the distances between the top and bottom planes was further calculated, and traditional measurement methods were used to measure the real distance inside the calibration box. Thus, the scale factor The experimental scenario for quantitative evaluation using the in vitro pre-calibration method is shown in We chose a particular site on the simulator organ’s surface as the target lesion and measured its dimensions as an evaluation metric. We selected five map points at both ends of the target lesion and performed five trials. In the real cavity environment, some organs or soft tissues often have uneven, wrinkled and irregular surfaces, which are more complex in spatial shape. Therefore, in the simulation of the cavity, the organ model shown in The 3D reconstruction effect of soft tissue after restoring the correct scale size by using the two methods of intracavity calibrators and in vitro pre-calibration is shown in It can be seen that the overall shape is consistent with the real model, and the contours and concave-convex details on the model are also clear, which can allow us to obtain the soft tissue information in the cavity well. The feature points of the local lesion were selected to evaluate the matching degree of the feature structure size, and to calculate the error between the reconstruction measurement results and the actual values.The measurement data and statistical error analysis results of the 3D reconstruction of the cavity are shown in By using two different models with different appearance morphologies and features to simulate the target organs and soft tissues in the cavity environment, the monocular SLAM in the cavity successfully realized the 3D reconstruction of the targets in both situations. For the quantitative evaluation of the system reconstruction results, two methods of in-cavity calibration and ex vivo pre-calibration were used to evaluate the reconstruction accuracy from the perspective of feature structure size matching. According to the error statistical data of the two experimental results of simulated organs and simulated soft tissues, it can be seen that the measurement error rate of both methods was around 10%, but the reconstruction model accuracy after restoring the real scale by using the in-cavity calibration template to provide prior knowledge was higher. Therefore, for judging the accuracy of the reconstruction results of the cavity SLAM system in a minimally invasive surgery scenario, introducing feasible calibration objects in the cavity first, such as small flexible rulers, surgical tools and other objects with obvious features and known size to be placed around the target, should be considered for data collection and reconstruction together. But when limited by objective factors such as surgical conditions, and relevant calibration objects cannot be placed in the cavity environment, then the method of in vitro pre-calibration should be chosen, so that the laparoscope can obtain the scale factor during the process of inserting into the cavity.In view of the degree of ambiguity of monocular SLAM and the difficulty in obtaining the actual true value of the lumen inspection data, which makes it impossible to quantitatively evaluate the accuracy of the iterative reconstruction of the lumen SLAM through common methods, this paper, starting from the direction of local coincidence, and based on the idea of GCP, designed two quantitative accuracy verification schemes for the reconstruction of intracavity calibrators and in vitro pre calibration. In addition, a simulated intracavity experimental platform was set up for actual measurement. The measurement results of the intracavity calibration reconstruction method in the two schemes were good, and the root-mean-square deviation of the reconstruction measurement in simulated organs and simulated soft tissues was 2.13 mm and 1.94 mm, respectively. Therefore, the feasibility of lumen SLAM and accuracy quantitative evaluation scheme was verified through the actual test of the simulated lumen, and the evaluation results can be used as a reasonable basis to measure the quality of iterative reconstruction of lumen SLAM.


In this paper, a multi-stage deep learning blood pressure prediction model based on imaging photoplethysmography (IPPG) signals is proposed to achieve accurate and convenient monitoring of human blood pressure. A camera-based non-contact human IPPG signal acquisition system is designed. The system can perform experimental acquisition under ambient light, effectively reducing the cost of non-contact pulse wave signal acquisition while simplifying the operation process. The first open-source dataset IPPG-BP for IPPG signal and blood pressure data is constructed by this system, and a multi-stage blood pressure estimation model combining a convolutional neural network and bidirectional gated recurrent neural network is designed. The results of the model conform to both BHS and AAMI international standards. Compared with other blood pressure estimation methods, the multi-stage model automatically extracts features through a deep learning network and combines different morphological features of diastolic and systolic waveforms, which reduces the workload while improving accuracy. With the progress of technology and social development, people’s living standards are gradually improving, while the morbidity and mortality of cardiovascular diseases are increasing year by year . CardiovBlood pressure (BP) is the lateral pressure acting on the blood vessel wall per unit area during the circulation of blood through the blood vessels around the body and is essential for maintaining blood circulation. As one of the important parameters to measure the health of the human body, it is not only used to monitor the health condition of the human body but also can be used for disease diagnosis and treatment, which has important observation value. One of the most important physiological parameters in the cardiovascular response to the state of the human body is arterial blood pressure, and its accurate, long-term dynamic monitoring assumes an extremely important role in the modern medical system . The comA non-invasive optical detection technique for physiological signal detection, called photoplethysmography (PPG) , has emeIPPG is a non-contact optical detection technique . When a The periodic signal, like IPPG, cannot be recognized by the naked eye, but the face video can be captured by imaging devices such as industrial cameras, webcams, and cell phone lenses, from which the IPPG signal can be extracted. In addition, because different components within the blood have different light absorption properties, the selected specific wavelength of the applied light source can achieve the measurement of different physiological parameters, such as heart rate , blood oCurrently, many scholars have conducted some preliminary exploratory experiments on non-contact blood pressure (NCBP) measurement ,14,15. SDeep neural network-based blood pressure estimation algorithms, which are implemented by feeding features or waveforms to neural networks, are one of the main directions of current research on continuous noninvasive blood pressure monitoring. Compared with machine learning-based measurement methods, deep learning models are more capable of learning for high-dimensional features and have a better fitting ability for complex nonlinear relationships. Some studies have used deep neural networks (DNN) to achieve blood pressure measurement through features extracted manually from pulse waveforms . HoweverCurrent machine learning-based methods are highly accurate and fast, but require manual feature extraction, and inaccurate feature extraction can seriously affect the measurement results for noisy signals. Neural networks have been proven to perform well in many fields and have been applied to blood pressure estimation, however, due to the simplicity of the pulse waveform, current studies are still unable to accurately apply valid waveform features, and their results are less accurate and more laborious.In order to overcome the above-mentioned problems, we investigated a deep learning-based algorithm for non-contact blood pressure estimation. A camera-based non-contact IPPG signal acquisition system is designed, and the first open-source IPPG signal and blood pressure dataset are constructed by capturing video of multiple volunteers’ faces as well as synchronized blood pressure data, and a multi-stage deep neural network is designed for blood pressure estimation to correlate the dynamic relationship between systolic and diastolic blood pressure to finally achieve z-accurate blood pressure measurement.(1)Use an external camera to capture the face video in a stationary state with stable ambient light and no other interference.(2)Extract the IPPG signal from the region of interest of the face video. Face recognition and key point location are performed on the frame images, and the ROI region with the most suitable effect for extracting IPPG signal is selected through comparison experiments. The three channels of RGB are separated from the frame images in the ROI region, and then the mean value of the pixel intensity of the green channel in the ROI region is calculated frame by frame, and finally, a fixed-length pixel intensity change curve is obtained as the original pulse wave signal and saved.(3)Signal pre-processing. The extracted pulse wave signal is wavelet transformed to remove the baseline drift, and then Butterworth bandpass filter is used to remove the high-frequency noise caused by the device itself to make the whole waveform smoother.In order to achieve better face video recording and signal acquisition, this project has developed a camera-based IPPG signal acquisition system, which is capable of capturing the images captured by the camera and displaying them to the system interface synchronously to facilitate the adjustment of the pose during recording and adding a timer recording function to meet the needs of different video recording durations with convenient operation. The interface of this acquisition system is shown in Most of the current studies on non-contact blood pressure measurement use expensive industrial cameras or high-definition digital cameras, which leads to high experimental costs, is not conducive to equipment deployment and product promotion, and cannot meet the demand for non-contact blood pressure measurement in daily home life. The earliest common camera-based physiological parameter detection method was proposed by Poh et al., in 2010 , and altThe laptop model used for the experiments is Legin R7000 2020.(1)Processor: AMD Ryzen 5 4600H with Radeon Graphics  3.00 GHz.(2)RAM: 16 GB.(3)Operating system: Windows 10, 64 bit.When the face video acquisition is completed using the camera, a series of operations such as ROI region selection, IPPG signal extraction, and processing need to be completed on the computer terminal.The blood pressure collection device used for the experiment was the Yuyue YE670A. The sphygmomanometer uses an LCD LCD soft backlit screen to clearly identify blood pressure readings and uses a SOC chip and upgraded DFFA dual filtering algorithm for more accurate measurement results, in line with ESH European Society of Hypertension and AAMI American Association for the Advancement of Medical Devices standards.When the video recording is completed, the system will first perform face recognition on the video, and if the face cannot be correctly recognized, the face video will be re-recorded; if it can be correctly recognized, the captured video will be saved, and used in the subsequent research. The algorithm used in this system is the Dlib face recognition algorithm, which has high accuracy and real-time performance and is a face recognition tool based on C++ and can be used in Python. The system uses a loaded Dlib face recognition model to locate face key points on separated frames. The model locates 68 key points, including organs such as mouth, nose, eyes, and the overall face contour. The face size can be calculated from the location information of the key points and the localization of the ROI region is completed.The IPPG technology is developed based on the Lambert–Bier law and light scattering theory ,28. LambThe IPPG signal is obtained by a weighted average of the pixel values in the area. The calculation formula is shown below:There is a lot of noise in the original IPPG signal, mainly including the power frequency noise of the equipment and the baseline drift caused by the movement of the face. In this experiment, the denoising operation is realized by wavelet transform and Butterworth bandpass filter, and the noise in the signal is removed.The wavelet threshold denoising method mainly analyzes and utilizes the information of the respective signal characteristics of the actual signal and the interference noise and performs the denoising process based on the different characteristics of the two when wavelet decomposition is performed. The noise reduction can be achieved by separating the effective signal from the noisy signal while maintaining the original characteristics of the actual signal to the maximum extent possible, and without destroying the signal needed for the experiment. The discrete wavelet transform of a one-dimensional signal is shown in Equation (3):The wavelet transform can be decomposed to different degrees according to the different characteristics of the signal to be denoised, in order to achieve the best signal denoising effect. In the practical application of signal denoising, the selection of wavelet transform scale is related to the sampling frequency.In this paper, the IPPG signal is taken as an example to perform six-scale wavelet decomposition, and the frequency ranges of each scale decomposition are shown in Studies have shown that the frequency range of the pulse wave is 0.7 Hz–6 Hz . For theFor the selection of ROI regions, some studies choose the overall face region for the experiment. However, parts including the eyes and corners of the mouth can produce small motion artifacts due to breathing or subconscious movements. Such motion artifacts can affect the experimental results to a certain extent. Therefore, it is not advisable to select all the face areas. It is also considered that when conducting experiments on female subjects, many people’s hair will cover the forehead area, which will also have an impact on the photography. A large number of studies have shown that only some facial regions contain rich vascular information. Since the selection of ROI is directly related to the accuracy of subsequent signal extraction, it is extremely important to select the correct ROI containing rich vascular information for the study. Therefore, this section selects the ROI region with the best signal extraction effect by conducting comparison experiments with IPPG signals extracted from three regions: forehead, nose and cheeks, and mouth, respectively.The experiments were conducted by selecting 8 face videos of the subjects, extracting IPPG signals from three different ROI regions, drawing the mean green channel grayscale curves of different regions, and finding the most suitable ROI region through comparison experiments. The video recording duration is 60 s, the resolution is 1920 × 1080, the frame rate is 60 FPS, the natural light irradiation conditions are carried out, and the experimental subject is required to be in a calm state when recording. The pulse wave signals were extracted from three parts of the forehead, nose and cheeks, and mouth, respectively, for comparison experiments. From the above figure, it can be seen that the signal waveforms extracted from the cheek and nose regions are smoother and have less noise interference compared with the other two regions. There are two reasons for this result: firstly, this region is more stable and does not produce larger displacement due to human respiration, so there is less interference from motion artifacts; secondly, this region is rich in capillary distribution, which can better reflect physiological information. Therefore, this region was selected as the ROI region for extracting IPPG signals in all subsequent studies.To ensure the accuracy of the experiment, a total of 115 volunteers aged 18 to 40 years old  from the School of Physics and Electronic Information of Zhejiang Normal University and Zhejiang Yunpeng Technology Co. were recruited to participate in this study. All volunteers understood the experimental task description in advance and signed the informed consent form before the experiment, and none of them took cardiovascular disease-related drugs or had any known cardiovascular disease.The frame rate of the face video recorded in this experiment was 60 FPS, and the image resolution was 1920 × 1080. Usually, the human heart beats at 0.5–4 Hz per minute, so the frame rate of the camera used in the experiment fully met the demand. Before recording the video, the volunteers were asked to sit quietly for five minutes to maintain a stable physiological state, and then sit on a chair about 0.5 m in front of the camera in an environment with a stable light source to record the video. In order to ensure the reliability of the experiment, two time periods were chosen: morning and afternoon. A schematic diagram of the data acquisition process is shown in https://github.com/czh846567921/IPPG-BP/tree/main, accessed on 28 February 2023). The dataset contains 2120 pulse wave segments and their corresponding blood pressure data. To facilitate model training, the study classifies the blood pressure data into one category for every 5 mmHg, and the histogram of the blood pressure data distribution is shown in During the face video recording, the volunteers were asked to keep their heads as stable as possible and not to wear accessories that would obscure the facial region of interest to ensure that the pulse wave signals could be extracted correctly. The study also used a commercial sphygmomanometer, the Yuyue YE670A (which meets ESH as well as AAMI standards) to measure blood pressure and heart rate simultaneously and record them, with the signal pre-processing process as described previously. Finally, by collecting data from 115 volunteers, we constructed and open-sourced the first dataset for imaging-based photovolumetric pulse wave signals and simultaneous blood pressure—IPPG-BP . The Softmax function is used as the activation function, the multivariate cross entropy  is used as the loss function for training, and the learning rate is set to a fixed value of 0.001. The structure of the BiGRU network is shown in In this project, we used Anaconda for experimental simulation, 70% of the data were used for training and 30% of the data were used for testing, we used TensorFlow deep learning framework to build the network model, we used the default Adam as the optimizer, the data padding method was Padding = same, we also added a Dropout layer to prevent model overfitting, the parameter setting was 0.3. The first-order CNN model is trained with 500 iterations and the batch size is set to 32. The second-order BiGRU model is trained with 500 iterations and the batch size is set to 32, and the experiments are repeated several times for each model, using different random seeds each time to ensure the randomness of the data. The random seeds ensure that the ratio of the training set to the test set is 7:3 for each training, the experimental subjects in the training and test sets are independent of each other, and finally, the average of all predictions is taken as the final result. Finally, the performance is evaluated on the test set.To ensure the accuracy of the experiments, a typical CNN model and a BiGRU model are also developed as comparison experiments on this topic. All these models have similar settings to the multi-stage model. To verify the accuracy of the models, three evaluation metrics, mean absolute error (MAE), standard deviation (STD), and mean deviation (ME), are used in this paper. In addition, the experiments also use regression curves and Bland–Altman plots for consistency analysis and regressivity analysis. Finally, the experimental results of this study were also compared with other related works. All analytical results below are based on data from the 30% validation set.In this experiment, the experimental results of the two-stage model were compared with the multi-stage model. From As can be seen from the table, the accuracy of the multi-stage model was higher than that of the separate-stage model. This indicates that fusing the dynamic relationship between systolic and diastolic blood pressure can effectively improve the accuracy of the blood pressure estimation model. The experimental results using the multi-stage model were analyzed subsequently.In this section, the experiments compare the prediction results of the multi-stage blood pressure model with two international metrics, the British Hypertension Society (BHS) and the American Association for the Advancement of Medical Devices (AAMI). The BHS classification criteria are based on the proportion of different mean absolute error ranges, which classify blood pressure measurements into three classes, A, B, and C, with error ranges of <5 mmHg, <10 mmHg, and <15 mmHg. The AAMI international standard differs from the BHS based on the mean deviation ME and standard deviation STD. AAMI requires that the ME and STD of DBP and SBP are less than 5 mmHg and 8 mmHg, respectively. From This section analyzes the prediction results of the multi-stage model through regression curves and Bland–Altman analysis plots. A scatter plot of the estimated blood pressure and the reference blood pressure is shown in Bland–Altman analysis plots can be used to detect the consistency of the data. The Bland–Altman analysis plots of estimated blood values and reference blood pressure are shown in A large number of scholars have conducted research on blood pressure estimation based on pulse wave signals. Li et al.  trained Blood pressure estimation based on pulse wave features requires manual definition and calculation of feature values, and the dramatic changes in IPPG signals caused by vascular diseases, aging and physiological activities can affect the accuracy of manual feature extraction, or even prevent the features from being extracted correctly. The multi-stage blood pressure estimation model based on deep neural networks proposed in this study uses two convolutional neural networks in the first stage to extract the local morphological features of each IPPG segment and provide a preliminary estimate of blood pressure. Convolutional neural networks (CNNs) originate from the connectivity pattern between neurons in the visual cortex and use convolutional operations instead of the usual matrix multiplication. CNNs have fewer parameter connections to facilitate training than standard fully connected neural networks with the same number of layers  and are The second stage is two BiGRU networks, which are used to train the temporal features of the pulse wave signal. GRU can be considered a variant network of LSTM. The main improvement is the use of a new structure called “update gates” to replace the input and forgetting gates of the original LSTM. By optimizing the network structure, the training time is reduced without changing the memory function of neurons. The role of the update gate is twofold: first, to determine how much of the previous information is passed to the later neurons; second, to determine how much of the previous and current information is passed on. The reset gate, on the other hand, is responsible for determining how much of the past information is forgotten. The update gate also determines how much of the new information and how much of the old information is retained. The old and new information are in a complementary relationship; the more new information is retained, the less old information is retained, and vice versa. Additionally, BiGRU can establish the relationship between the information of the current moment and the features of the previous moment as well as learn the association between the feature information of the future moment and the information of the current moment. The BiGRU network consists of two GRUs with opposite directions, so BiGRU can transmit information both forward and backward so that the feature association between the future information and the past information is established, which solves the time series data features that are forgotten, thus improving the accuracy of blood pressure measurement. Notably, the study fully considers the dynamic relationship between systolic and diastolic blood pressure and establishes the hidden relationship between different features and blood pressure.In this study, a camera-based non-contact facial signal acquisition system is designed. The system uses an external camera to record facial video, uses the Dlib algorithm to dynamically locate key points of the face, and selects regions of interest that contain rich vascular information so as to extract IPPG signals in ROI and preprocess them by wavelet transform and band-pass filtering. The first open-source dataset for IPPG and blood pressure signals, IPPG-BP, containing 115 volunteers with 2120 signal segments and corresponding systolic and diastolic blood pressure records, was constructed by a self-designed non-contact facial signal acquisition system. A multi-stage deep learning blood pressure estimation model based on CNN and BiGRU was designed. The multi-stage model uses a deep learning network to directly extract features from the pulse wave signal, solving the problem that traditional manual extraction of features is susceptible to interference from vascular diseases, aging, and physiological activities. The physiological relationship between diastole and systole is also considered, and the initial blood pressure estimates obtained using the diastolic and systolic feature vectors are then cross-fed into the BiGRU network in the second stage for data expansion, respectively. Experimental results show that the model can accurately achieve blood pressure estimation. The model also requires only a single IPPG signal to be acquired for blood pressure estimation, which is easily accessible and reduces workload while improving accuracy.


We introduce a technique to generate and read the digital signature of the networks, channels, and optical devices that possess the fiber-optic pigtails to enhance physical layer security (PLS). Attributing a signature to the networks or devices eases the identification and authentication of networks and systems thus reducing their vulnerability to physical and digital attacks. The signatures are generated using an optical physical unclonable function (OPUF). Considering that OPUFs are established as the most potent anti-counterfeiting tool, the created signatures are robust against malicious attacks such as tampering and cyber attacks. We investigate Rayleigh backscattering signal (RBS) as a strong OPUF to generate reliable signatures. Contrary to other OPUFs that must be fabricated, the RBS-based OPUF is an inherent feature of fibers and can be easily obtained using optical frequency domain reflectometry (OFDR). We evaluate the security of the generated signatures in terms of their robustness against prediction and cloning. We demonstrate the robustness of signatures against digital and physical attacks confirming the unpredictability and unclonability features of the generated signatures. We explore signature cyber security by considering the random structure of the produced signatures. To demonstrate signature reproducibility through repeated measurements, we simulate the signature of a system by adding a random Gaussian white noise to the signal. This model is proposed to address services including security, authentication, identification, and monitoring. C) ends with the individual output function called response (R). New challenges will not end with the same response, and every single challenge has its own unique response. The significant fact about PUFs is that any PUF material has a unique function, which means applying the same challenge to two PUF materials results in two different responses. This feature allows PUFs to be considered a strong security technique and a good alternative to today’s techniques based on cryptography. It is worth mentioning that the term PUF includes several technical and technological categories. Among them, electrical PUF has been considered in [Communication around the interconnected world is an inseparable part of quotidian life. Indeed, fiber-optic communication networks are the most deployed telecommunication infrastructures across the globe . Modern dered in  as a PLSdered in . Opticaldered in  and for dered in .In this paper, we introduce a practical technique to enhance network PLS based on OPUF. The fundamental feature of our proposed technique is the identification of networks, network elements, optical links, systems, and sub-systems, that possess optical fiber or fiber pigtail. The proposed technique is investigated using the following procedures: In the first step, the optical signature of the device under test (DUT) is read and stored. The signature is generated using the Rayleigh backscattering signal of the fiber, which is considered a strong OPUF . In thisThe main idea of this work is to exploit the intrinsic physical features of fiber-optic communication systems to supplement their security against adversarial attacks. Generating and assessing a signature for every fiber link or fiber-optic device included in the network provides a higher level of security for the system. Accordingly, any network or its fiber-optical constituent element can be identified and authenticated. Thereby, any attack, whether physical or digital, can be detected. Physically invading the system will cause a change in the system signature that could confirm the presence of an imposter. Even digital invaders with advanced machine learning (ML) knowledge cannot predict the system’s signature  because Although in fiber-optic systems, RBS is usually considered a disruptive noise that leads to throughput degradation, we propose to exploit RBS to read the signature of fibers. Any fiber has an individual RBS pattern. This is a result of the randomly distributed particles along the fiber core, which occur during the manufacturing process of the fiber. These particles are much smaller than the wavelength of the light used in the communication system that propagates into the fiber and causes Rayleigh scattering. The unknown position and various distribution of the particles in the fibers is an advantage that causes the RBS-based signature to be unpredictable.Optical time or frequency domain reflectometry (OTDR/OFDR) is generally used to measure fiber RBS. From the large pool of proposed OFDR techniques in the literature ,34, we ci-th reflected point and round-trip time of reflected light at each point, and The received RBS combined with LO at the photodetector  can be eThe RBS-based signatures rely on the OPUF challenge–response protocol. Thus, any stimulus  maps a unique result  and provides a challenge–response pair (CRP). C): the frequency-modulated continuous wave (FMCW) parameters on the Tx side represent the challenge. FMCW parameters are: sweep rate Challenge (ified in . The reqR): the obtained RBS pattern represents the response. By changing FMCW parameters i.e., changing the challenge, the RBS pattern will change accordingly. Thereby, we can extract different RBS patterns from the same fiber if we stimulate the fiber with various challenges. Due to this fact, even if the adversary has access to the fiber, he cannot generate the intended signature without applying the correct challenge.Response  code, i.e., a two-dimensional (2D) binary matrix, to denote the signature. This process has been performed through the MATLAB code by converting bits into the binary image. In the worst-case scenario, the adversary has physical access to the nominated fiber (or device) and possesses strong knowledge and high-performance devices that can predict the key. In this case, the adversary cannot find out the signature without knowing the key parameters. Consequently, adding a key with the mentioned parameters to the original data enhances the signature robustness against such adversary attacks. In We propose a new method to generate a signature associated to any optical fiber segment  to assist services including security, identification, authentication, and monitoring at the physical layer. However, the generated signatures must be unique and reproducible for every fiber. Thus, repeating measurements using the same challenge should result in the same signature. Moreover, it should be robust against physical and digital attacks. To address these significant features, some specific metrics must be considered.(A)Cross-Correlation Coefficient (XCOR-C):A and B are binary signatures with shows how distinguishable the generated signatures from a single fiber (OPUF) are with different challenges;shows that noisy signature is distinguishable from an imposter version which is generated with the same OPUF but with the wrong challenge;assists in finding the best challenges to generate more reliable signatures.In ,35,36,37The uniqueness assessment of the signatures is implemented through the XCOR-C. In this study, we have considered the Pearson cross-correlation coefficient, which can provide information regarding the structural similarity between the acquired signatures and accurately distinguish the noisy signature from the imposter. It is the most common way to assess the similarity between acquired PUF responses .(B)Euclidean Distance (ED):The ED between the RBS obtained through several measurements of the same OPUF and the same challenge translates into reproducibility;ED between the RBS obtained through several measurements of the same OPUF but different challenges indicates unpredictability;ED between the RBS obtained through several measurements of various OPUFs but with the same challenges indicates unclonability.(C)Hamming Distance (HD):HD between the binary sequence achieved through several measurements of the same OPUF and the same challenge indicates reproducibility;HD between the binary sequence achieved through several measurements of the same OPUF but different challenges indicates unpredictability;HD between the binary sequence achieved through several measurements of various OPUFs but with the same challenges indicates unclonability;HD between each row of the binary image (QR code) gives uniqueness ,38,39.It is worth mentioning that the results of HD (ED) will commonly be demonstrated in histogram form . In this(D) Statistical Test Suite (STS)h = 1 or h = 0 with the probability of randomness p. If h = 0, the bits in S are in random order with the probability of p. Small values of p cast doubt on the validity of a null hypothesis. When h = 1, the test rejects the randomness of bits.The aim of using STS is to run a test to investigate the probability of randomness of the bits in the signature sequences (S). Wherein, the hypothesis test result is returned as h = 0 verifies the results in [One of the most significant ways to explore a signature’s cybersecurity is by considering the random structure of the produced signatures (bits). In , we demosults in .To validate the security robustness of the signatures generated using our proposed model, in terms of uniqueness, unclonability, and unpredictability, we considered various simulation scenarios for a 0.5 m fiber pigtail using C-OFDR. The measurement scenario was modified depending on which security feature of the signature was to be explored. In all scenarios, we considered a single polarization signal measure with C-OFDR.To investigate the reproducibility and unpredictability of the signatures, which are defined in the following section, we considered an ADC with N = 4000 samples.Ci . The signature was stored as valid. Subsequently, we generated 100 signatures of the same fiber with the same challenge (Ci) adding random white Gaussian noise to emulate the signature robustness against measurement noise during different times of measurements. We implemented ED between the original signature and the noisy ones, as shown in To validate robustness, we generated the signature by applying the following challenge: To validate unpredictability, we generated the signature of the same fiber by applying 40 different challenges by changing the sweep rate (and sweep range in a few cases). One worst-case scenario of an adversary attack is when the adversary has access to the fiber and attempts to use different challenges to obtain the original signature. In essence, any new challenge generates a new response, even though the challenge is applied to the same PUF, and this fact neutralizes the adversarial attempts. To demonstrate that, we implemented ED between the original signature and the signatures obtained with different challenges. The results are depicted in To validate unclonability, we generated 100 signatures from the 100 different fibers by applying the same challenge to all of them. Another worst-case scenario of adversary attacks is when the adversary knows the challenge but has no access to the original PUF and tries to apply the known challenge to different PUFs to find the signature. Because any PUF has a unique response for each specific challenge, the adversary trials will fail again. We implemented ED between the original signature and the other signatures obtained from other fibers with the same challenge as the original signature. In all cases, the ED was implemented before the RBS conversion to the binary domain. To investigate the robustness, unpredictability, and unclonability of the signatures with the HD metric, we converted the abovementioned signatures to the binary domain, added the key, and generated the QR code. The normalized histogram of HD (expressed in percent) between the original signature and noisy signatures, and between the original signature and the signatures obtained with different challenges, is shown in The HD between the original signature and noisy signatures, and HD between the original signature and the other signatures obtained from other fibers with the same challenge as the original signature are shown in Are noisy signatures adequately distinguishable from imposters?To answer this question, we studied signatures through the XCOR-C metric. The original signature (QR code) was generated and stored in the database. The XCOR-C was, then, calculated between the original and noisy signatures. Meanwhile, different signatures (imposters) were generated by applying various challenges to the same OPUF . Subsequently, the XCOR-C between the generated signatures and the original signature was calculated. The obtained results indicate that different challenges on one OPUF make different signatures distinguishable from the original signal with the noise, as shown in Afterward, we dropped one noisy signature among the imposter signatures. Once more, we carried out the XCOR-C. The original noisy signal (genuine) was detected and appeared with the highest cross-correlation coefficient (peak) among the results. The results are shown in By regarding the obtained results, it is possible to define a threshold (TH) to accept or reject the signature as genuine. Consequently, the decision rule will be: if XCOR-C is below a certain threshold (here 0.5), the signature will be considered an imposter, but if it is above the TH, it will be deemed genuine.One of the most significant ways to explore a signature’s robustness against cyber attacks is to consider the random structure of the produced signatures (bits). Indeed, a sequence that consists of random bits is extremely strong against cyber attacks attempting to predict that sequence. To investigate the randomness of our proposed signatures, we generated 500 signatures (QR codes) with 2D and 60 × 60 bits, generated by 3500 samples and a key of 100 bits. After running the STS test among 500 signatures, only seven of them failed the test, i.e., 1.4% failed sequences. We increased the signature dimension by measuring the RBS with a higher number of samples (4000) and achieved a signature with 64 × 64 bits, consisting of a 96-bit key. The test showed only 0.6% failed sequences. Eventually, we increased the number of samples (5000) to increase the QR resolution to 71 × 71 bits, consisting of a 41-bit key. As a result, we obtained 0.0% failed sequences i.e., all signatures passed the test. The probability of randomness of signatures with different QR resolutions is depicted in Optical identification (OI) through an optical signature is a perfect security complementary tool to the networks and guarantees the security of systems, sub systems, and devices. Such an OI identification method may find uses within several applications. It can be used in applications related to classic network security , quantumTransmitter Signature, Channel Signature, and Receiver Signature, respectively. In this scenario, when the channel is supposed to be a passive sub-system, three possible security approaches can be envisaged:Channel Signature and Receiver Signature to be sure that the information will pass to the specific channel and reach the specific receiver;the transmitter reads the Transmitter Signature and Channel Signature so that the receiver knows the sender and the physical path;the receiver reads the Transmitter Signature so that both know with whom they are talking. Both may even check for Channel Signatures to check the path.the transmitter reads the receiver, and the receiver acquires the In this paper, the concept of a network digital signature has been introduced, in particular, it has been proposed to generate such a signature by resorting to using optical unclonable functions (OPUF) that are already intrinsically present in fibers and determined by the specific internal characteristics of each fiber. Specifically, the OPUF that has been individuated and then tested in this work is the Rayleigh backscattering signal (RBS). This kind of signal has been analyzed in order to understand if it is a good OPUF candidate that satisfies all the requirements that a PUF, in general, must fulfill to be adopted as a security mechanism in authentication scenarios. Such requirements are usually unclonability, robustness, unpredictability, and randomness; all of them have been thoroughly verified and evaluated on the basis of different metrics such as Hamming distance (HD), Euclidean distance (ED) and cross-correlation coefficient (XCOR-C). In all the different cases, the RBS has been demonstrated to significantly perform and has been confirmed to grant both robustness and distinctiveness.The optical identification concept represents a novel approach to physical layer security which can be applied to any optical communication system and network. It provides supplementary quality and security to optical systems and network operations.Future works will be dedicated to further investigating other typologies of metrics specific to real-valued signals, such as RBS, to better understand if superior performances can be achieved, particularly in terms of robustness. Following this line of investigation, diverse operative scenarios will be considered to comprehend the actual applicability of the proposed solution in more depth.


Industrial data scarcity is one of the largest factors holding back the widespread use of machine learning in manufacturing. To overcome this problem, the concept of transfer learning was developed and has received much attention in recent industrial research. This paper focuses on the problem of time series segmentation and presents the first in-depth research on transfer learning for deep learning-based time series segmentation on the industrial use case of end-of-line pump testing. In particular, we investigate whether the performance of deep learning models can be increased by pretraining the network with data from other domains. Three different scenarios are analyzed: source and target data being closely related, source and target data being distantly related, and source and target data being non-related. The results demonstrate that transfer learning can enhance the performance of time series segmentation models with respect to accuracy and training speed. The benefit can be most clearly seen in scenarios where source and training data are closely related and the number of target training data samples is lowest. However, in the scenario of non-related datasets, cases of negative transfer learning were observed as well. Thus, the research emphasizes the potential, but also the challenges, of industrial transfer learning. The ubiquity of sensor data in modern manufacturing is creating opportunities as well as challenges for manufacturers. Since both computer-aided engineering  and timeA lesser-known research area is time series segmentation (TSS), which deals with splitting up a time series into distinct non-overlapping segments. In a manufacturing setting, TSS is usually based on multivariate time series data. TSS can be utilized to detect different operational states of production machines or to recognize human actions via tracking sensors. Multivariate time series data pose special challenges for algorithms. The most successful multivariate TSS approaches proposed in recent years are based on deep learning and use a supervised learning concept . In recThe biggest drawback of deep learning models is their heavy dependency on the availability of data. In particular, in many industrial environments, the collection and labeling of required data is expensive and time consuming. In turn, there are many settings where only a limited amount of training data are available. This widespread issue of data scarcity led to the emergence of research fields that aim to find approaches able to cope with a small amount of training data. Examples of corresponding approaches include few-shot learning, data augmentation, and transfer learning (TL). The concept of TL is illustrated by source dataset before transferring the learned parameters to a different model, the target model. Subsequently, the target model is retrained by the target dataset, which often comprises a small number of training samples (finetuning). As a basic idea, the information that the model learned by solving the source problem might be helpful to boost the training process of the target problem. Especially for production plants, TL constitutes a promising approach, as a high number of different product variants and models with widely varying quantities may be manufactured on the same shop floor. In particular, the data from products manufactured in high quantities might provide interesting information for similar goods with less turnover. This idea was recognized by many manufacturers, triggering a high number of publications in recent years about the successful use of TL in industrial settings . Partexample, . In geneWe analyze the benefits of using TL for TSS in an industrial setting. The paper provides one of the very first works to even tackle the problem of TL for TSS in general.source datasets with varying degrees of similarity to the target dataset affects the performance of the target model after finetuning.We systematically analyze how pretraining with three different target dataset.We analyze to what degree the benefit of TL depends on the amount of available samples in the The use case analyzed in the paper deals with the segmentation of operational states within the end-of-line testing cycle of hydraulic pumps. This is an innovative application of time series-based deep learning for a practical manufacturing problem.With the presented problem setting and identified research gap in mind, this paper provides the following contributions:The remainder of the paper is structured as follows: Facilitated by the general breakthrough of deep learning in many domains, the related research field of TL has recently received much attention and has advanced significantly. Different surveys were published in recent years, providing a detailed overview of the state of the art. The most extensive one focuses on the general concept of TL over all domains, including experimental results on text categorization and object detection . More resource datasets affects the classification accuracy of the target task )source tasks and target tasks investigated in this paper belong to the category of supervised TSS tasks. More precisely, we target to find and train a deep learning model to enable it to predict the state labels for all timestamps of a multivariate time series. This is called dense labeling. The network architecture we use to analyze the impact of TL in this work is called PrecTime. The PrecTime architecture was introduced in 2022 in one of our previous works [All us works . It is dus works . They wesource datasets . Thus, a far lower number of samples is required to achieve good results compared to other deep learning tasks. The three settings are explained in detail in the following section.The experimental setup consists of three settings with varying sets see . The souSetting 1 (Same-Asset Pretraining): Source data closely related to target datasource datasets and target datasets are taken from the HPEoL dataset. For the experiment, all three subsets belonging to the DC pump type are selected, with each of the subsets representing the 40 samples of a distinct DC-pump version . The technical design of the three pump versions differs only slightly. In turn, the testing cycles of the three pumps are very similar as well. Only detailed specifications of certain testing phases may vary. Based on the high similarity, we call this setting same-asset pretraining. Using the formalization introduced in X and label space Y of source data and target data are identical, while the probability distribution target dataset, while the data of the remaining two versions are merged to become the combined source dataset. For each of these three scenarios, we perform various experimental runs, in which we limit the sample number in the target training set to 1, 3, 5, and 10. This allows us to investigate how the number of samples in the target training dataset affects the benefit of pretraining considering training speed and accuracy. Furthermore, two layer freezing approaches are compared. In the first approach, all layers except the output layer were frozen (TL-fr). In the second approach, all layers were set as trainable (TL-tr). A non-pretrained model that was trained only by the target dataset is defined as the baseline (BL1). Additionally, as the feature space X and label space Y are identical, an additional baseline can be created by evaluating the model performance when merging the data of all three pump versions into one big training set (BL2). Note that this additional baseline is only available in Setting 1, as it requires equivalence of feature space and label space.In the first setting, both Setting 2 (Cross-Asset Pretraining): Source data distantly related to target datasource datasets and target datasets are subsets of the HPEoL dataset. However, in this case, all three pump types  are considered. Each of the three DC pump versions  is used as the target data, while both the SC pump subset and the PC pump subset are distinctly used as source data. The testing cycles of the different control types  differ heavily. However, the used test benches and sensors are identical across all types. Therefore, we have still have an identical feature space X across source data and target data in this setting. We limit the target domain training data to three instances to make the impact of TL strongly visible. The TL is performed with all layers unfrozen. A non-pretrained model is set as the baseline (BL1).As in Setting 1, all Setting 3 (Cross-Domain Pretraining): Source data non-related to target datasource dataset can boost the accuracy of the target task. As in Settings 1 and 2, the three DC pump versions are distinctly chosen as target sets. In this case, however, the pretraining is not done by HPEoL subsets, but by the Opportunity dataset . For the Opportunity dataset only the drill-section data were used as samples, and 113 sensors were selected as is the common practice in the literature [The networks were trained on an ting see . The traterature . MissingThe following section gives detailed insight into the experimental results and also discusses them.source dataset from the same asset type increased the final accuracy of the network in all experiments. A clear negative correlation between the amount of available target data and the benefit of TL can be observed, meaning TL shows its greatest potential when facing target data scarcity. While the exact numbers differ in detail, the observations are consistent across all three DC pump versions. When having a target training set of size one, the accuracy difference between the unfrozen TL-tr approach and the isolated training baseline (BL1) ranged from 10% for the V38 pump to 34% for the V35 pump. On the contrary, in the experimental runs using the highest number of target training samples (10), the performance boost decreased to around 2–3%, which is helpful but far less significant. Interestingly, no clear conclusion could be drawn about the benefit of combining target data and source data into one large training set and training the network with it (BL2). While in some experimental runs, the combined training led to accuracy improvement compared to isolated target data training (BL1), in other cases it seemed to impede model performance. However, note that combining source data and training data into one big dataset (BL2) in no case showed a better performance than the TL-tr approach, emphasizing the superiority of using TL in this setting.target training set of size 3. Clearly, we can observe that for unfrozen TL (TL-tr), not only is the final accuracy improved (higher asymptote), but also the training speed is significantly increased. Very high accuracies (>90%) are reached far earlier compared to the two baselines . The main reason for the increased training speed seems to be not a steeper slope, but a higher starting point achieved through pretraining. The extent of this observation varies across the three pumps; it is most visible in the V35 training process. The high starting points after pretraining seem to be the result of using source data and training data of high similarity. For frozen TL, it becomes obvious that freezing all hidden layers almost completely eliminates any training progress in this setting. The models end up with almost the same performance before and after finetuning (TL-fr).source data from a different asset type), the benefits of TL are diminished clearly compared to Setting 1  at the start of the training.However, when taking a closer look at the training process in source dataset (Opportunity) in this setting belongs to a completely different domain than the target dataset (HPEoL), pretraining does not lead to any benefit, but instead impairs the training speed of the target data  is similar to or only slightly below that of the non-pretraining scenario (BL1) for all three pump variants. This can be explained by the fact that all layers were set to trainable. Thus, the pretrained models are able to overcome their impaired starting point caused by cross-domain pretraining over the course of the training process and fully adapt to the new domain.As opposed to the observed benefits of TL in Settings 1 and 2, we can observe a form of negative TL in Setting 3. As the data see c. The ditarget training data and (2) increasing the similarity between source data and target data. Altogether, the findings provide systematic evidence for the hypothesis that the benefit of TL in TSS is (1) increasing with lack of While freezing layers is often recommended in the literature, in this work, it heavily aggravated the model performance in initial experiments (see TL-fr results in Setting 1) and was therefore not investigated deeper. However, checking different layer freezing strategies in a more nuanced way might be an interesting direction to go for future research.source datasets improves the network learning performance of PrecTime, a sequence-to-sequence deep learning architecture for TSS. Focus points were the speed of the training process and the achieved validation accuracy after completing the training. The target task was specified to be the supervised segmentation of hydraulic pump testing cycles into different testing phases via the prediction of state labels. Three different source datasets were selected for pretraining (1) different versions of same pump type, (2) different pump types, and (3) different domains. It was observed that pretraining with a closely-related source dataset led to the biggest benefit (Setting 1), while pretraining with a source dataset from a different domain even aggravated the training process (Setting 3). Additionally, we could show that TL performed best in comparison to baselines in scenarios in which the target training dataset had the smallest sample size.We performed a systematic investigation of different TL scenarios and strategies for an industrial TSS problem. It was examined whether and how pretraining with close- to non-related The main limitation of our study is the focus on a specific use case. While our results provide interesting findings, the domain-independence and general validity of the results still have to be established. Furthermore, the important topic of layer freezing was not investigated in depth. Therefore, future work could look deeper into different layer freezing strategies and aim to provide recommendations as to which layers should be frozen and which layers set to trainable. Additionally, one could check if incrementally unfreezing layers over the course of the training process leads to promising results. Possible research directions could also include performing experiments with different TSS network architectures and datasets to validate and build on the findings of this paper. It could also be examined how pretraining with multiple datasets or a general-domain dataset affects the model performance. Finally, the transfer of knowledge between labeled data and unlabeled data could be investigated in the context of TSS.