In the face of cooperative intelligent transportation systems (C-ITS) advancements, the inclusion of vulnerable road users (VRU), i.e., pedestrians, cyclists, and motorcyclists, has just recently become a part of the discussion. Including VRU in C-ITS presents new challenges, most notably the trade-off between the increase in VRU safety and the aggravation in channel congestion resulting from VRU-generated messages. However, previous studies mainly focus on network-related metrics without giving much consideration to VRU safety-related metrics. In this context, we evaluated such a trade-off with a study of motion-based message generation rules for VRU transmissions. The rules were analyzed using theoretical and simulation-based evaluations. In addition to studying the message generation rules using channel load metrics, such as channel busy ratio (CBR) and packet delivery ratio (PDR), we introduced a new metric: the VRU Awareness Probability (VAP). VAP uses the exchange of messages from active VRU to measure the probability of VRU detection by nearby vehicles. Results show that fixed message-filtering mechanisms reduce the overall channel load, but they could negatively impact VRU detection. We established the importance of quantifying the VRU awareness and its inclusion in C-ITS analysis because of its direct impact on VRU safety. We also discussed approaches that include VRU context and dynamism to improve the definition of message generation rules. Vehicular networking represents one of the critical elements to enable a diverse set of applications related to intelligent transportation systems (ITS). Such applications range from safety to convenience services bringing benefits to accident prevention, accident mitigation, traffic optimization, comfort, and many others .In the safety area, several approaches have been proposed to detect (mobile) road users other than vehicles. The most common approach is passive detection, in which vehicles use different sensor technologies, such as video analysis and object detection based on non-visible light, to characterize the surroundings  and to identify possible traffic threats involving other road users. Nevertheless, the need for line-of-sight (LoS) restricts the proper operation of the sensing devices in the passive approach. Moreover, because the LoS condition cannot be ensured in all situations, the passive approach is unsuitable for several safety use cases, as previously pointed out by many authors ,4,5,6,7.Despite the benefits that C-ITS brings to traffic safety, existing systems still fail to include all possible road users as active participants in the communication processes . One relIn recent years, several efforts have sought to reduce road traffic deaths and improve the safety of VRU using communication technologies. These efforts involve exchanging messages between vehicles and VRU using vehicle-to-pedestrian (V2P) communications; V2P includes the exchange of information between vehicles and any type of VRU. Different technologies are proposed for V2P, including Wi-Fi , cellulaIn vehicular networks, VRU can take on active or passive roles. An active VRU participates in the communication process by sending information , usually in a periodic manner, or following a particular set of transmission rules. A passive VRU waits for messages sent by vehicles and may reply only when it receives a message. Establishing the suitability of these roles for the effective integration of VRU in C-ITS is still an open challenge ,16. The The congestion problem becomes even more critical when considering the radio access technology. For example, in the case of DSRC, there are concerns about the suitability of the technology to support the increase in network traffic associated with VRU. These concerns are mainly because this technology already suffers from network congestion on the Control Channel—used for the exchange of safety packets—when the network density increases ,5,19. ThDifferent proposals of decision systems address network congestion problems associated with the active participation of VRU. Typically, they manage the transmission rate or filter the transmissions of active VRU according to the context or motion characteristics. Such context-aware systems use environmental variables and VRU motion features  to modify the transmission rate of VRUs’ mobile devices. In general, the proposed message generation rules allow the overall reduction in network load measured by the channel occupancy ,5,19.Although the use of rules achieves a network load reduction, a crucial factor that has not been sufficiently evaluated is the ability of contextual-decision systems to maintain VRU awareness when applying message filtering. We define VRU awareness as the proportion of VRU detected by a vehicle over the total number of VRU nearby. We maintain that this parameter is critical in contextual-decision systems because of its direct relation to the capacity of vehicles to detect possible threats and avoid traffic accidents through the exchange of messages. Understanding the trade-off between reducing network congestion and maintaining an appropriate level of VRU awareness in contextual-decision systems is what motivated this work.We defined a novel VRU awareness metric to analyze the impact of VRU message generation rules on the timely detection of VRU. Our proposed VRU awareness probability metric (VAP) measures the ability of a vehicle to detect the surrounding VRU as a function of the probability of successful message reception.We conducted a trade-off study based on channel busy ratio (CBR), packet delivery ratio (PDR), and VAP to analyze VRU message generation rules proposed in the literature . For thiWe developed a simulation tool to validate our theoretical model, which differs from previous works and tools that consider only pedestrians and cars e.g., ,5,19). I,19. I5,1In an effort to improve the safety of VRU, this work investigated the capacity of vehicles to identify nearby active VRU using contextual systems that apply message generation rules. The contributions of this paper can be summarized as follows:The remainder of the paper is organized as follows. Although contextual-based transmission mechanisms are a well-investigated field in vehicular communications, to date, similar techniques that focus on VRU do not present the same depth of exploration. Elements of vehicular context-based triggering systems were used as an important inspiration for this work. One example is the development of the context-based rules for collective perception messages (CPM) and the evaluation of the effectiveness of detection using such messages (closely related to awareness). CPM, standardized by the European Telecommunications Standards Institute (ETSI) , allow tThe context-based rules include contextual variables of the detected objects, such as their movement, novelty, and speed changes to determine if a message is transmitted. Previous works have studied the channel load variations, derived from the transmissions triggered by these rules compared with a baseline case—a fixed message transmission rate. They also introduced metrics related to object awareness. In , the autreceive-only mode, where mobiles are only allowed to receive. However, such a mode resembles a passive approach, which can reduce pedestrian detection compared to an active communication approach [In the context of VRU communications, the authors in  developeapproach .Sewalkar et al.  studied Lee et al.  investigBagheri et al.  proposedRostami et al.  studied The study scenario includes nodes of both types: VRU and cars equipped with IEEE 802.11p connectivity . In addiAlthough the assumption that pedestrians, cyclists, and motorcyclists may be equipped with a communication unit that supports the IEEE 802.11p is not yet a reality, there have been initiatives for implementing a DSRC protocol stack in smartphones , with noThe sets The proposed system model presents the message generation rules that govern VRU transmissions, together with the 802.11p model used to derive the evaluation metrics of the system. The notations used in this section are summarized in The generation of safety messages by VRU follows each one of the three rules introduced in  (see belPedestrian on Street (PedOnStreet): Pedestrians only transmit messages when they are on the street. Other types of VRU like bicycles and motorcycles—grouped under the name cycle—are assumed to always be on the street, so they always transmit messages under this rule. A representation of this rule behavior is shown in Moving VRU (MovVRU): VRU transmit only when they are moving (VRU’s velocity is different from zero); thus, stationary VRU do not transmit.Multiple Transmission Rates (MultiTx): This rule modifies the transmission rate , and they are the same for all nodes; (3) data packets arriving for transmission at a node follow a Poisson distribution, with the same parameter R; and (6) a highway scenario is assumed.For the theoretical analysis of the message generation rules and calculation of metrics, we adapted the IEEE 802.11p model proposed by Hassan et al. . The autAlthough we focus on an urban scenario, we justify using the model in  since thTo calculate the probability of a busy channel, thors in  employedof pb in  is:(2)pbT is the packet transmission time, including the wait for a distributed coordination function (DCF) interframe space, DIFS. P), C), and In this equation, Equation , there iEquation  is the nEquation  is the nHere, we present the equations used for the computation of the collision probability and the probability of a node transmitting in an arbitrary time slot. As we consider arbitrary tagged nodes in each category of road users, the collision probability of the jth node category can be expressed as follows:The probability of any transmission in an arbitrary time slot, defined as To evaluate the generation rules, we propose a novel awareness metric, the VRU awareness probability (VAP), to account for the knowledge that vehicles have about VRU presence in their surroundings. Such a metric is complemented with the channel busy ratio (CBR) and the packet delivery ratio (PDR) to account for the channel load and successful delivery of packets, respectively.To calculate CBR, we employ Equation . In the Equation  for the VAP assumes that VRU are only detected through the exchange of messages. Under this definition, the metric represents the proportion of correctly received VRU messages over the total number of VRU nodes in a tagged vehicle’s transmission range . VAP is PDR was defined as the probability of the successful delivery of messages from a tagged node. For VAP computation, we considered that PDR is the same for every node; hence, the probability of the correct reception of VRU messages at the tagged vehicle is equal to the probability of effective delivery as seen by each VRU node that surrounds the tagged vehicle under study.i). To calculate VAP can be expressed as follows:shown in , the augTo support the inclusion of active VRU in the wireless communication network, we developed an extension to the Veins framework  inspiredWe focused on evaluating an urban intersection scenario in which VRU naturally co-exist and interact with vehicular traffic. In particular, the buildings and roads were extracted from the simulation presented in , and VRU2). In both scenarios, the number of pedestrians is considerably larger than the other nodes to better observe the impact of the message generation rules related to moving/stationary pedestrians. The two scenarios used for simulations are presented in We extracted a portion of the map in , correspFor the definition of the scenarios, we used as boundaries the density values used by Rostami et al. . We compTo study the performance of the different evaluation metrics, we simulated the scenarios using four message transmission rates: 1 Hz, 2 Hz, 5 Hz, and 10 Hz. The first three transmission rates were previously tested by . The ratWe performed a comparative analysis of the theoretical model introduced in We evaluated the theoretical model using Matlab. In the model, we assumed a scenario geometry that approximates an intersection in a square grid of 200 m by 200 m to make it consistent with the simulations. It also considered the transmission power and sensibility threshold of the simulations, according to the values presented in After obtaining both numerical and simulation results, we compared the results of CBR and PDR to validate the precision of the model against a realistic scenario such as the one employed in the simulations. The results for the comparison of CBR and PDR are presented in This section evaluates the impact of the message generation rules that control the sending of VRU context-awareness messages. First, we obtain the baseline results, where VRU send messages at a fixed rate . We then analyze the metrics considering the use of the message generation rules. In both analysis , we consider the same two scenarios and density compositions as shown in In the following step, we study the variations of the node behavior. Hence, we vary the proportions of moving pedestrians and cycles, as well as the percentage of pedestrians on the street. Such a sensitivity analysis is only performed with the theoretical model, given the limitations of modifying the proportions of nodes at will in the simulation tool because it uses realistic traces .First, we analyze the performance of the baseline rule in the two scenarios defined in Our proposed awareness metric (VAP) provides additional, useful information when the frequency is varied. In the low-density scenario e, we obsVAP variations can be explained by observing the formulation in Equation . VAP hasHere, we present and discuss the results obtained for CBR, PDR, and VAP for the low and high-density scenarios when the message generation rules are in place. The x axis presents the data rates used by VRU to transmit when using each rule. When the MultiTx rule is in place, there is no dependency on the frequency; hence, it is labeled as ribed in .In terms of channel load, The other metric that was used to quantify channel load was PDR. We present PDR results in Despite the improvements observed in terms of channel load, VRU awareness demonstrates a different behavior. VAP results are presented in The VAP decrease is less critical under the MovVRU rule, showing a variation of about 10% when compared to the simulated baseline. A higher number of moving VRU explains this smaller decay. The case of the MultiTx rule is interesting in terms of awareness analysis. We conclude that message generation rules can have different and opposite effects on channel load and VRU awareness. This is shown most notably with the PedOnStreet rule, where the most significant decrease in the channel load came with the most considerable decrease in VAP compared to the other rules.Using the theoretical model see , we perfFirst, we present the results when applying the PedOnStreet rule. In this case, the moving variable is the percentage of pedestrians on the street; cycles are assumed to be always on the street. We vary the percentage of pedestrians on the street between 0% and 100%. From Despite the improvements in PDR and CBR, VAP decreases. For example, in the high-density scenario b, while In contrast with the previous rules, the evaluatio of the MultiTx rule  differs We chose DSRC as the access technology for the current study mainly because of its extended use and research. Other access technologies of interest, such as C-V2X, may vary on their levels of PDR, CBR, and VAP; however, we believe that our proposed analysis can apply to C-V2X technology with minor modifications to the metrics’ measurements. Despite the difference in the level of congestion and the awareness probability between the two technologies, the results, particularly in PDR and VAP, should follow the same trend ,41. The Among the message generation rules under study, the MultiTx is the rule that shows the best performance in terms of the trade-off between congestion—measured through CBR and PDR—and awareness (VAP). The MultiTx rule varies VRU transmission frequency between 2 Hz and 5 Hz when the VRU are stopped or in movement, respectively see . These aAlthough the new combination of frequencies have an impact on PDR and CBR, VAP appears almost insensitive to the frequencies variations in the low-density scenario see c. In theases see e, VAP shases see . Figure This section presented a study of a set of message generation rules based on channel load (CBR and PDR) and VRU awareness capabilities (VAP). As demonstrated in In the following, we summarize our findings based on the results obtained studying the rules proposed in . Simple With respect to context, the most common feature is to defer transmission when pedestrians are detected to be indoors; indoor sleeping mode was already considered in  and the With respect to the VRU intention, several contributions propose to use visual sensors (mainly cameras), thermal sensors, RADAR (radio detection and ranging) technology, and LiDAR (light detection and ranging) to determine the intention ; the focWith respect to dynamic rules, we consider that the dynamic nature of the vehicular environment requires context-aware and adaptive rules. This is the subject of our current research; more specifically, we are developing a decision system, based on machine learning techniques, that allows to dynamically select between DSRC and C-V2X (cellular–vehicle-to-everything) in a heterogeneous network.One of the main obstacles to the correct deployment of VRU systems is channel congestion. Several message generation rules have been proposed to cope with this problem. However, despite the benefits of this kind of rules in channel congestion, the effect on VRU detection has not been previously studied. In this paper, we introduced a novel VRU awareness metric—VRU awareness probability (VAP)—and studied existing message generation rules on channel load and VRU awareness. We performed the study using realistic simulations in Veins and theoretical models implemented in Matlab. Using these tools, we performed an analysis with real mobility traces and a sensibility study based on the variation of rule activation. To the best of our knowledge, the study of message generation rules with a scope including VRU awareness capabilities is new in the context of VRU active communications.From our analysis, we observed that despite the benefits that message generation rules can bring in terms of channel congestion relief, there may be adverse effects on VRU detection capabilities mainly due to the lack of transmission in the presence of specific motion characteristics. We also showed that the degradation in detecting VRU through communications could be even more critical in scenarios with large proportions of VRU that do not satisfy the transmission criteria of fixed rules.Explore different access technologies: The use of other access technologies is an open field of research. One example of these technologies is C-V2X, a technology developed by 3GPP specifically designed for V2X communications. Works on C-V2X have shown improvements compared to DSRC ; howeverMultiple access technologies: The use of different access technologies in a heterogeneous architecture could also be considered for channel load reduction without necessarily reducing VRU awareness. Current access technologies, such as DSRC, WiFi, and C-V2X, could coexist in the vehicular environment. There is a number of contributions ,18,29,50Position-based rules: Another line of research is the exploration of message generation rules that combine the position and direction of VRU with the motion-based policies to filter unnecessary messages. For example, rules could stop the transmissions of pedestrians considered to be in a safe zone  or walkClustering: As indicated by ETSI , the cluOverall, we consider our main contribution is to call attention to the importance of including VRU awareness analysis when formulating message generation rules in C-ITS systems. We defined a new metric that can support this additional evaluation. Finally, we proposed research directions to meet the challenges of adding VRU signals to the communication mix. In the following, we present a list of these challenges:


The SuperCam instrument suite onboard the Mars 2020 Perseverance rover uses the laser-induced breakdown spectroscopy (LIBS) technique to determine the elemental composition of rocks and soils of the Mars surface. It is associated with a microphone to retrieve the physical properties of the ablated targets when listening to the laser-induced acoustic signal. In this study, we report the monitoring of laser-induced mineral phase transitions in acoustic data. Sound data recorded during the laser ablation of hematite, goethite and diamond showed a sharp increase of the acoustic signal amplitude over the first laser shots. Analyses of the laser-induced craters with Raman spectroscopy and scanning electron microscopy indicate that both hematite and goethite have been transformed into magnetite and that diamond has been transformed into amorphous-like carbon over the first laser shots. It is shown that these transitions are the root cause of the increase in acoustic signal, likely due to a change in target’s physical properties as the material is transformed. These results give insights into the influence of the target’s optical and thermal properties over the acoustic signal. But most importantly, in the context of the Mars surface exploration with SuperCam, as this behavior occurs only for specific phases, it demonstrates that the microphone data may help discriminating mineral phases whereas LIBS data only have limited capabilities. Quantification of elemental chemistry may be achieved for major and some minor elements after careful calibration of the instrument. The breakdown process occurs when the laser heats, melts then vaporizes the matter. Therefore, LIBS is destructive and any molecular and mineralogical structure information is lost during the ablation process. Thereby, LIBS is not able to discriminate between two different mineralogical phases with the same chemical composition but with a different structure such as mineral polymorphs. When performed under a surrounding gas, i.e. not under vacuum, the plasma expansion generates a shock wave that relaxes into an acoustic wave. This wave carries rich information that can be recorded and analyzed by a microphone2. The energy released within the shock wave actually depends on several target properties such as the optical absorption and the thermal dissipation4. Moreover, recording the laser sparks has been shown to provide data complementary to the LIBS optical spectrum and key information such as the target’s hardness and the volume of the laser-induced ablation cavity6.Laser-Induced Breakdown Spectroscopy (LIBS) is an analytical technique that relies on the spectral analysis of the photons emitted by a plasma plume excited by a short and energetic laser pulse focused on a solid target. The spectral analysis of the plasma light allows to identify the atoms that compose the ablated matter8, that combines color imaging, LIBS, time-resolved Raman spectroscopy and visible and near-infrared reflectance spectroscopy. This remote sensing instrument includes a microphone used to complement the LIBS investigation of SuperCam, and to record the atmospheric noise in the audible bandwidth9. Perseverance is exploring the geology of the Jezero crater and one main scientific objective is to detect possible traces of ancient life on Mars10. Perseverance will also cache the first samples for return to Earth by future missions. In that context, SuperCam provides key chemical and mineralogical information on the rocks for in situ science and to optimize the selection of the samples.On February 18, 2021 NASA’s Perseverance rover landed in Jezero Crater, Mars, carrying onboard the SuperCam instrument suite8. It may also alter locally the mineralogical structure within this laser-induced crater12. For example, some high optical absorption minerals have been shown to exhibit laser-induced mineral phase transitions due to melting followed by recrystallization (hematite is transformed to magnetite for instance) or amorphization (graphite into amorphous carbon) within the LIBS crater11. Such transitions were completely achieved after 30 shots, for a laser energy comparable to SuperCam. The formation of a new mineral phase or material inside the laser-induced crater has strong implications as the targeted spot may have a different chemical composition as well as new physical properties. Hence, it likely induces some changes in the laser-matter coupling. Consequently, one could wonder whether this change may lead to a modification of the acoustic signal during the shots when the transition takes place.To perform remote LIBS analysis of surrounding rocks up to 7 m from the rover, SuperCam uses a telescope to focus the high-irradiance laser and collect the signal subsequently analyzed by a spectrometer. Irradiance deposited on the target  versus high (opacity) optical absorption of the LIBS infrared laser beam respectively, and extreme hardness for diamond. Moreover, diamonds might eventually be found in Jezero crater due to impact shock metamorphism of carbonates in the margin of the crater. Indeed, diamonds are found in Ureilites meteorites, likely formed due to impact shock15.The goal of this study is to investigate whether an acoustic monitoring of LIBS on some specific mineral phases can highlight some phase transitions and/or transformations induced by the laser ablation. To do this, iron oxides were selected to test this hypothesis: hematite , its decrease rate is larger compared to magnetite. On the other hand, hematite, goethite and diamond exhibit a sharp increase of the acoustic energy during the The shot-to-shot evolution of the acoustic energy for bursts of 30 shots for the various minerals is presented in Fig. This increase in the acoustic energy for these three targets may be indicative of a change in the laser-matter interaction over the first 5 shots and, therefore, of a potential transformation of the target induced by the laser. Hence, craters obtained from 1 to 5 shots on a crystal of hematite, goethite and diamond are further characterized by Raman spectroscopy and SEM to elucidate the cause of the increasing acoustic energy recorded by the microphone data.SEM images of the crater resulting from 1 laser shot are shown in Fig. In contrast, most of the crater floor is composed of a rough, cracked layer of molten metal that appears to slightly cover the pristine hematite because it has similar roughness  over a depth between 200 nm and 18. Under Mars atmosphere, the melting temperature of both hematite and magnetite can be estimated using the temperature-pressure diagram calculated for the Fe-O system19. Assuming that the maximum oxygen partial pressure corresponds to the maximum possible 11, such a temperature was likely exceeded in the LIBS crater. In any case, the quench of the melt leads to the crystallization of magnetite and not hematite as predicted by the temperature-pressure diagram calculated for the Fe–O system19. This magnetite remains stable within the layer covering the LIBS crater and does not back-transform into hematite as observed by Raman spectroscopy, even after a single LIBS shot. The molten layer is very thin at the surface of hematite, about 200 nm11. The very fast cooling favors the crystallization of submicrometer-sized magnetite crystals. Interestingly, the thickness of the magnetite layer at the surface of hematite is large enough to explain why the Raman only probes magnetite and not the underlying hematite. Indeed, the high absorption optical coefficient of magnetite at 532 nm20, i.e. about The observations of the laser-induced craters show that the laser has strongly affected the mineralogical structure of the hematite and goethite. Some experiments and existing models of temperature rise within hematite21. For the target, both its thermal and optical properties govern how efficient is the laser-matter interaction. The optical penetration depth 22. Thus, ablation is less efficient for long penetration depth. It was also shown that the surface aspect is a key parameter that controls the laser-matter interaction: steep walls receive a lower irradiance compare to planar horizontal surface5.Therefore, one could wonder why an increase of the laser-induced acoustic signal is observed on hematite and goethite, which both suffer a phase transition, whereas such an increase is not observed when firing the laser on a pure magnetite crystal see Fig. a,b. LaseAfter only one laser pulse on the hematite and goethite, the crater floor is partially transformed into the molten-like magnetite. Therefore, the subsequent pulses are fired on this new phase that has different properties compared to the initial pristine mineral . Consequently, over the 5 first shots on hematite, the increase of the magnetite coverage in the crater explains the increase of the laser-induced acoustic signal. Similarly, the increase of the laser-matter interaction over the five first shots leads to a more luminous plasma and a stronger LIBS optical spectrum.Furthermore, in the case of the hematite, the roughness of the initial pristine material explains that, in some part of the targeted surface . Therefore, it may contribute to progressive loss in laser-matter coupling.On the other hand, no remarkable phenomenon is observed on the acoustic signal recorded from graphite Fig. c whereas11, the present study shows that it can also be instantaneously and remotely recorded by acoustic data. For the minerals experiencing a laser-induced phase transition, the laser-induced acoustic signal shows an increase of its amplitude over the first shots. It correlates with the formation of a new phase within the ablation cavity: magnetite for hematite and goethite, and amorphous-like carbon or locally very disordered graphitic carbon for diamond. The increase in the acoustic amplitude is explained by a change in the optical and/or thermal properties of the samples, as it is transformed by the laser. It consequently modifies the laser-matter interaction and then the strength of the shock-wave.Laser ablation transfers a huge energy density into the targeted material and this study showed that it locally affects the mineralogical structure of three of the samples analyzed: hematite, goethite and diamond. While this phenomenon was already described, especially on hematiteAlthough diamond is a less relevant target for Martian surface exploration, its study allows us to better understand the influence of the physical properties of the samples that drive the laser-induced acoustic signal, with a simple example. In particular, it is shown that the optical absorption coefficient plays a major role in the amplitude of the acoustic signal. Indeed, this is the first time that the physical properties of a sample  are used to explain the behavior of the acoustic signal. More generally, this study shows how the microphone is an aid for mineralogy through the signatures of the physical properties of minerals.32. Thus, LIBS cannot detect the stoichiometric variations between the compositions of various iron oxides. In addition, LIBS cannot distinguish the degree of oxidation of these iron oxides based on the LIBS active spectrum, and thus distinguish between magnetite and hematite33. On Perseverance, the SuperCam instrument combines a LIBS experiment and a microphone. Given the results of this study, the observation of an increase in the LIBS acoustic signal amplitude on a pure iron oxide target would indicate the likely presence of hematite or goethite. Furthermore, the observation of the hydrogen line in the LIBS spectrum can help concluding between goethite and hematite. Of course, this acoustic method will be complementary to the other mineralogical investigations provided by SuperCam, e.g. visible and near-infrared reflectance spectroscopy. The Raman signal from these opaque iron oxides is very weak, and so, SuperCam does not have a chance to detect them with time-resolved Raman spectroscopy.On the other hand, the acoustic detection of phase transition on iron oxides has a major implication for the exploration of the Mars surface chemistry with LIBS. In Gale crater, almost pure iron oxide nodules are observed with the LIBS experiment of ChemCam. However, LIBS on Mars gives limited information on oxygen, as the oxygen signal in the LIBS spectrum originates from both the sample and the breakdown of the atmospheric carbon dioxide34. Indeed, such a mineral has a large optical penetration depth and therefore, is less likely to experience a sharp temperature rise, which may prevent any phase transition and acoustic effects. Previous observations of LIBS craters on some low optical absorption hydrated samples, like gypsum or hydromagnesite, with Raman spectroscopy11 did not show any apparent dehydration or structural modification, at least over the entire volume excited. Indeed, for these kinds of mineral, the Raman signal comes from a relatively large volume and may not be fully representative of the very first nanometers at the surface. Consequently, more experiments on hydrated minerals with a different analytical strategy are needed to highlight the specific cases where laser-induced dehydration occurs, and if it can be systematically tracked with acoustic data.As for the goethite, this is the first observation of a mineral being dehydrated by LIBS. Therefore, one could wonder how it could be extrapolated to other hydrated minerals relevant to Mars. Previous experiments to track laser-induced dehydration on hydrated samples led to inconclusive results, except for gypsum for which no significant dehydration was observed35, microphone may become a critical add-on to such instruments to provide key information on the physical properties of the samples analyzed.As the laser-induced breakdown spectroscopy becomes a new standard for planetary exploration36 . Due to an idiosyncrasy of the telescope of this setup, the ablation crater is composed of two lobes, which can be seen in Fig.  The LIBS capability of this setup used the ChemCam Mast-Unit Engineering and Qualification Model that deposits about 10 mJ on a target through a 5 ns infrared pulse 1067 nm). The laser was fired at a frequency of 3 Hz. The Mast-Unit was coupled with the ChemCam Body-Unit Engineering Model that includes three spectrometers collecting the radiations from the plasma over the UV (240.1 nm to 342.2 nm), the violet (382.1 nm to 469.3 nm) and the visible plus near infrared . The laser beam was redirected into a vacuum chamber that contains the samples. The chamber was filled with a controlled Mars atmosphere . This configuration yielded a horizontal resolution of Scanning electron microscopy (SEM) analyses were performed using a Zeiss Ultra 55 field emission gun SEM. Secondary electron images were acquired using an in-Lens detector (for nanotopography of the sample) at an accelerating voltage of 5 kV, a working distance of  Minerals were obtained from the Collection de Minéralogie at Sorbonne Université . These are natural single crystals. The provenances are: hematite , magnetite , goethite , graphite (Madagascar) and diamond .


The dynamics of increased intracranial pressure (ICP) and sixth cranial nerve palsy has undergone a paradigm shift, with emphasis shifting from a length hypothesis to a theory based on novel anatomic findings pertaining to the geometry of Dorello's canal. In particular, the sixth cranial nerve resides in a transfixed coaxial cylinder within the canal. The cisternal portion of the nerve is intradural and the rest of the nerve is extradural; therefore, with increased ICP, the former is stretched, thereby pulling on the rest of the nerve, which is anchored in Dorello's canal. We present a case of pseudotumor cerebri secondary to minocycline presenting with an isolated sixth nerve palsy. This case is used as a platform to segue into the recent findings outlined above, in particular, the evolutionary transformation of Dorello's canal from a circular outline with a bony roof to an elliptic profile with a fibro-osseus roof during hominid basocranial expansion. The fibro-osseus roof, being elastic, is particularly susceptible to the influence of raised ICP, thereby narrowing the canal and injuring the sixth cranial nerve. The sixth cranial nerve is quite unique: a pure motor nerve, the second longest cranial nerve after the trochlear nerve, entirely dedicated to supplying the lateral rectus muscle, thereby allowing abduction of the eye. Its course at the base of the brain is remarkable - after leaving the ponto-medullary junction, it enters the subarachnoid space , and after a variable distance, it penetrates the dura , leaves the intramural space, and travels through the osteofibrous Dorello's canal under the petrosphenoidal ligament (PSL), also known as Gruber's ligament, stretching between the petrous apex (spine) and the accessory clinoid process (ACP) . While hDuring this treacherous journey, the sixth cranial nerve is highly vulnerable to injury, especially with high intracranial pressure. This was thought to be due to its long intracranial course, however, a study of 26 cadavers showed that the sixth cranial nerve is one-third the length of the trochlear nerve, and the former is more likely to be injured by raised ICP . The sixThe differential diagnosis of an isolated sixth nerve palsy includes ocular myasthenia gravis, thyroid ophthalmopathy, and microvascular disease from diabetes and hypertension. For the sake of completion, we will mention the brainstem syndromes of Raymond, Millard-Gubler, and Foville syndrome as they are not etiologies of isolated sixth nerve palsy and also involve other lower cranial nerves, gaze centers, and long tracts. Gradenigo's syndrome with petrous apex spread from otitis media via the inferior petrosal sinus involves cranial nerves V, VI, VII, and VIII, and is of historical interest .Lastly, as in our case report, a sixth nerve palsy can be an isolated finding in patients with pseudotumor cerebri/idiopathic intracranial hypertension and can be seen in up to 30% of cases. The injury is likely due to stretching of the sixth cranial nerve but is almost always reversible and therefore a neuropraxia .A 17-year-old young woman presented with an insidious onset of unremitting severe bilateral occipital headaches associated with a roaring sound in her left ear. The severity of the headache required her to seek care in the emergency room twice. One week into her illness, a fixed horizontal diplopia with maximal separation of images with left direction of gaze had developed, necessitating the use of an eye patch. She did not report any transient visual obscurations or other vision loss. Despite being overweight, no increase in weight was recorded over the previous year. Her past medical history was significant for acne, for which was treated with monocyline 100 milligrams (mg) daily for the last three months. Otherwise she was a healthy young woman.2. Gait cadence, tandem-walking, and heel-and-toe walking were normal. Ocular motion reveals a frank left sixth nerve palsy  scale was 5/5 throughout both upper and lower limbs. Deep tendon reflexes were lively in upper and lower limbs. Touch-pressure, vibratory, and joint position sense were normal in the fingers and toes.A magnetic resonance imaging (MRI) of the brain was normal with and without contrast. A magnetic resonance venogram (MRV) showed patent venous sinuses Figure .An MRI of the brain and orbits with and without contrast showed bilateral optic nerve sheath distention with flattening of the posterior bulge of the sclera (lamina cribrosa) consistent with optic disc edema Figure .A supine lumbar puncture in the lateral decubitus position revealed a high cerebrospinal fluid (CSF) pressure, greater than 55 centimeter (cm) of water pressure . CSF revealed normal protein and no pleocytosis. The patient was diagnosed with pseudotumor cerebri secondary to minocyline therapy, a well-established and tight association. Minocycline was discontinued and the patient was started on a slowly escalating dose of 250 mg of acetazolamide daily with a target dose of 750 mg daily, as tolerated. By the end of the first week, the headache had completely resolved, and by the end of the second week, the diplopia had resolved. At the two-month ophthalmologic visit, the optic disc edema had also resolved and the patient was weaned off acetazolamide.Dorello's canal, also known as the petroclival segment, marks the transition between the posterior and middle cranial fossa. This canal has undergone evolutionary change being fused superiorly by a bony arch in non-hominid primates, the union of the petrous apex (spine) and the ACP. With hominid brain expansion and growth of the basicranium, the non-union of the petrous apex and ACP was replaced by an elastic fibro-osseus ligament, the PSL, covering the roof of the foramen. The canal is elliptic in shape in humans and rounded in non-human primates, the expansion of the cranial base converting it from a circle to an ellipse Figure  10]..10].The elastic nature of the PSL allows high ICP to squeeze and compress the PSL and constrict the sixth cranial nerve in Dorello's canal leading to a sixth nerve palsy. Furthermore, the PSL is butterfly-shaped, with a narrowing or stricture in the middle in 78% of cases. Meanwhile, the sixth cranial nerve runs in the middle of the canal, where it is susceptible to compression by the vertical bending forces on the middle portion of the elastic PSL into Dorello's canal . In non-The cisternal segment of the sixth cranial nerve is intradural and therefore subject to raised ICP. Meanwhile the rest of the sixth cranial nerve, including the petroclival and cavernous segments, are extradural and not subject to raised ICP. Therefore, with a diffuse and isotropic increase in ICP, the nerve is subject to a differential pressure with the proximal cisternal segment of the nerve pulling or tugging on the rest of the nerve; this being a tentative explanation of a neuropraxia of the nerve. Contrast this to transtentorial herniation where there is global stretch of the nerve with descent of the pontomedullary junction and massive increases in ICP (plateau waves) and subsequent necrosis of the nerve due to stretching of the nerve beyond its yield point (Hooke's law) ,13.With Dorello's canal being defined as the space or trough between the clivus and petrous bone bounded superiorly by the the PSL, anatomic variations do occur and this space is often referred to as the petroclival region Figure .There are instances wherein the sixth cranial nerve courses over Dorello's canal and some authors have suggested an alternative appellation; the petroclival venous confluence (PVC). This region is akin to chamber within the confines of the meningeal and periosteal layers of the dura with the confluence of multiple venous sinuses draining into the inferior petrosal sinus . The PVCEvolutionary developmental biology or "evo-devo" is an exciting field that helps explain functional anatomy pertaining to evolutionary descent. The expansion of the basicranium parallels the growth of the brain during hominid evolution. The morphological changes are many, one of which pertains to a classic and frequent presentation in neurological practice: a sixth nerve palsy. Localization of a sixth nerve palsy has a well-established differential diagnosis and is an exercise of neurological finesse and beauty. However, classical neurologists have also noted the "false localizing sign" of a sixth nerve palsy in cases of diffuse and isotropic increase in ICP. Here there is no focal disease but a global process, which as a quirk of neuroantomy has predisposed this nerve to compression at Dorello's canal due to the transformation of the roof from an osseus structure to a fibro-elastic and compressible tent as a result of brain expansion during hominid evolution. This is the classic "false localizing sign" of a sixth cranial nerve palsy, which now has a plausible and beautiful explanation.


Manual visual inspection of civil infrastructure is high-risk, subjective, and time-consuming. The success of deep learning and the proliferation of low-cost consumer robots has spurred rapid growth in research and application of autonomous inspections. The major components of autonomous inspection include data acquisition, data processing, and decision making, which are usually studied independently. However, for robust real-world applicability, these three aspects of the overall process need to be addressed concurrently with end-to-end testing, incorporating scenarios such as variations in structure type, color, damage level, camera distance, view angle, lighting, etc. Developing real-world datasets that span all these scenarios is nearly impossible. In this paper, we propose a framework to create a virtual visual inspection testbed using 3D synthetic environments that can enable end-to-end testing of autonomous inspection strategies. To populate the 3D synthetic environment with virtual damaged buildings, we propose the use of a non-linear finite element model to inform the realistic and automated visual rendering of different damage types, the damage state, and the material textures of what are termed herein physics-based graphics models (PBGMs). To demonstrate the benefits of the autonomous inspection testbed, three experiments are conducted with models of earthquake damaged reinforced concrete buildings. First, we implement the proposed framework to generate a new large-scale annotated benchmark dataset for post-earthquake inspections of buildings termed QuakeCity. Second, we demonstrate the improved performance of deep learning models trained using the QuakeCity dataset for inference on real data. Finally, a comparison of deep learning-based damage state estimation for different data acquisition strategies is carried out. The results demonstrate the use of PBGMs as an effective testbed for the development and validation of strategies for autonomous vision-based inspections of civil infrastructure. The inspections of structures that are necessary after earthquakes are laborious, high-risk, and subject to human error. Describing the nature of inspections in a post-disaster scenario, the ATC-20 field manual  states tData acquisition with unmanned aerial vehicles (UAVs) and data processing using deep learning algorithms have shown tremendous potential in advancing the level of autonomy in post-earthquake inspections. A frequently studied problem is the application of machine learning algorithms, with a specific focus on the use of deep convolutional neural networks (CNNs) for damage identification after earthquakes. Yeum et al.  proposedThe significant progress in computer graphics software over the past decade has allowed the creation of photo-realistic images and videos from 3D synthetic environments that have spurred advances in computer vision. The data generated from these graphics models, termed synthetic data, provides the ability to rapidly generate a large amount of diverse data  that may be impossible to otherwise acquire in the real world. Synthetic data has been used to validate applications like robotic simulation  and forResearchers have recently begun utilizing 3D synthetic environments for applications in vision-based inspection and monitoring. Hoskere et al. ,32,33 prThis paper proposes a novel framework for automatically generating 3D synthetic environments spanning diverse scenarios in structure properties, loading, camera properties, and environment necessary for a robust inspection testbed. In particular, we propose a procedure to generate physics-based graphics models (PBGMs) that incorporate a finite element model with non-linear time history analysis for modeling the response of a structure, and novel graphics algorithms to render physically consistent damage. Another significant contribution of our research is the demonstration of the utility of the proposed testbed through experiments with reinforced concrete RC buildings subject to earthquake excitation. First, we implement the proposed framework to generate a new large-scale annotated benchmark dataset for post-earthquake inspections of buildings termed QuakeCity. Second, we demonstrate the improved performance of deep learning models trained using the QuakeCity dataset for inference on real data. Finally, a comparison of deep learning-based damage state estimation for different data acquisition strategies is carried out. A general flowchart of the testbed process is provided in 3D synthetic environments  are defiA framework for the generation of physics-based graphics models (PBGMs) for inspections is now presented. For clarity, the framework is illustrated in a schematic presented in The geometry of the structure of interest in the PBGM is represented by a 3D mesh. The mesh may be created in any 3D creator software. The features of buildings incorporated in the 3D mesh will enable networks trained on synthetic data to learn representations for those features in real images. For structural inspections of buildings, structural components like beams, columns, and shear walls, and non-structural components like infill walls, doors, windows, and balconies, are highly relevant as damage to these components provides visual indicators of structural health. Similar lists can be made for other types of structures to be inspected. All these components should be created programmatically through parameterization, or, as referred to in the field of computer graphics, created “procedurally”. Procedural generation of the mesh will allow programmatic implementation of subsequent steps, thus enabling randomization of both geometry and textures. Randomization has been shown to improve the performance for related tasks like robotic perception when learning from synthetic data by Tobin et al.  and is rFrom the perspective of PBGM generation, non-linear finite element analyses provide valuable insight into the regions in a structure where damage is most likely to occur. The same parameters used to construct the mesh procedurally are used to generate finite element models as well. In the particular case of post-earthquake inspections, a two-step analysis approach is proposed, first obtaining a simplified global response of the structure and then conducting a high-fidelity analysis for the visible components to generate accurate damage patterns. The main pieces of information derived from these analyses are the plastic strain contours, and other damage indicators such as the compression damage index from a concrete damaged plasticity model, which provides direct indicators for cracking and spalling of members–two of the main visual indicators of structural health after an earthquake. As the distribution of plastic strain is not likely to change for small changes in the loading, the number of analyses can be further reduced for large structures with little effect on the final result  by taking advantage of the fact that components often repeat in a structure . The next subsection describes the proposed methodology to identify physics-based damage hotspots using non-linear analysis. Damage masks are 2D binary images that indicate the presence of damage on component surfaces. Several damage parameters need to be determined before these masks can be generated using the conducted analysis. These parameters relate to the number, size, shape, and location of the damage. Each of the relevant parameters may be determined through, (i) physics-based response, or (ii) defined heuristics. Both these modes come with their own set of merits and demerits. Heuristic methods are the only viable option for many damage cases that are difficult to model  or for which no empirical data is available. Methods stemming from empirical data are reliable because they are based directly on observations but identifying good heuristics is challenging. When realizable, physics-based damage masks provide a rigorous approach that links the visual representation to results of finite element analyses, leveraging efforts by researchers in developing state-of-the-art constitutive models. Incorporating the physics enables applications such as estimating structural response , failure mechanisms, and model updating. We first propose a general framework to determine damage parameters and then demonstrate generating masks for common damage types of cracks and spalling using the structural response. DS) of the component based on some structural response measure DS and The damage parameters are obtained by Monte Carlo sampling from empirical or heuristic distributions. The first step is to determine the damage state  Bidirectional scattering distribution functions, (ii) material textures, (iii) damage textures, and (iv) annotation textures.Bidirectional scattering distribution functions: The visual appearance of an object is dependent on how light incident on its surfaces is scattered, transmitted, or reflected. In computer graphics, the behavior of light incident on a given material is represented by a bidirectional scattering distribution function or BSDF . BSDFs cMaterial textures: PBR textures encompassing maps with BSDF parameters for the base color, roughness, metallicity, etc., and surface normals can be used to adequately represent materials for the purpose of structural inspection simulation. PBR textures for common construction materials incorporating BSDF parameters created through height field photogrammetry are available on websites like CC0textures . A samplDamage textures: The damage textures for the PBGM are obtained using the material textures as the base and modifying the region within the generated physics-based damage masks using opencv-Python . The craAnnotation textures: For deep learning methods, the ground truth synthetic data is rendered by using an emission BSDF. As opposed to the principled BSDF with 10 parameters, an emission BSDF has a single color parameter and acts as a light source. The emission shader is useful for rendering homogenous colors, which is what is required as ground truth for tasks like semantic segmentation. Depending on whether image data or annotation data is being rendered, the appropriate texture types are selected during the rendering process.The generated textures are applied to the components after UV unwrapping the components. For 3D models application of 2D textures requires a correspondence to be created such that 2D surfaces can map to corresponding locations on the 3D surface. This process of “unwrapping” the 3D model is termed UV unwrapping. UV unwrapping is conducted by selecting the edges that are to serve as seams to break up the 3D model. In most programs, once the seams are selected, the resulting 2D surfaces are then arranged to fit within a square surface. The obtained damage masks are also assembled in the same arrangement to create a direct correspondence to the UV map and thus to the 3D model. Other masks like the rebar mask are also arranged in the same way. Here, depending on the aspect ratio of the component, the arrangement can take on a handful of configurations that are hard coded along with the dimensions of the corresponding component so that the rest of the process can be automated. An example of a UV unwrapped image is provided in The steps discussed thus far describe the construction of a single PBGM. To obtain photo-realistic images, the background scene also needs to be populated. For post-earthquake building inspections, which include multiple buildings, roads, sidewalks, light poles, electric cables, trees, etc. Randomization of geometry and textures is important towards the ultimate goal of the generalizability of deep learning models trained in the synthetic environment. Thus procedural methods are adopted even in the scene assembly for the generation of these items. The final step is to render the images. There are two modes of rendering commonly available in 3D creator software, namely path tracing and rasterization. Path tracing involves simulating the path of light in the scene and is more computationally expensive than rasterization but is preferred as it produces more photorealistic representations. To render images, a light source and the camera locations and orientations are to be set in the synthetic environment. To simulate realistic outdoor lighting, HDRI maps are used to light the scene . The proposed framework was implemented using multiple software applications. 3D model construction was conducted in Blender 2.9 , the fin3D models were created for 12 different fictional reinforced concrete buildings with masonry infill walls. The main reason for creating multiple buildings is to be able to generate diverse data that can be used for further experiments. The layout for these buildings was loosely 3 buildings shown in  that werThe dimensions and layout of the building were parameterized to include dimensions and locations of columns, beams, walls, windows, and balconies. The building properties were stored in a single class object that were used both for finite element model creation and 3D model generation. As mentioned in Reinforced concrete sections were created for the members using the patch, section, and layer commands in OpenSeesPy. The amount of rebar was set based on ACI 318-14  assumingThe shear reinforcement was assumed to be at a spacing of a maximum of  QuakeCity Dataset: Large-scale synthetic dataset of earthquake-damaged buildings, (ii) Augmenting real data with synthetic data, and (iii) Comparing post-disaster UAV data acquisition with ground camera data acquisition.Images are rendered from multiple simulated UAV surveys of 11 damaged buildings in a city environment to create a new dataset called QuakeCity. Each survey replicates a field scenario where a UAV circles the building at different altitudes to cover the entire height, width, and length of the building. Each image captured by the simulated UAV is associated with six different sets of annotations, including three damage masks , components, component damage states, and a depth map. In total, 4688 images and six annotations per image of size 1920 × 1080 are included in the dataset, with 3684 for training, and 1004 for testing. Example images of the generated dataset are shown in To reliably train an autonomous visual inspection system, a large amount of training data with damaged structures would be required. Frequently, however, the amount of such training data available is limited. Additionally, careful annotation of available images is also a challenge. In this experiment, we are interested in studying whether the incorporation of synthetic data in cases with limited availability of real data with annotations can help boost the accuracy of networks on unseen real data. A dataset for semantic segmentation of real earthquake-damaged buildings was developed for the purpose of this study. The images were acquired by the authors after the 2017 Mexico City Earthquake using a DJI Phantom 3, and a Nikon D3300. The images were annotated for the presence of spalling using InstaDam . In totaA deep network is constructed for semantic segmentation using a ResNet  architecEight different models were trained to evaluate the potential role of synthetic data in enhancing the overall performance of the models on real data. The eight models included four pairs of training schemes listed in The results from the different models trained are shown in The addition of synthetic data was also shown to improve the performance of the deep neural network even for varying splits of training and testing data. While implementing autonomous visual inspection systems after disasters, a trained model using a dataset conducted prior to the disaster would be used to process new data acquired after the disaster. The quality of the predictions on new data may however vary widely depending on the image acquisition distance. For example, it may not always be possible to have consistent data acquisition modes or distances for various structures of interest. This is especially so in crowded cities where many obstacles are present. To better study the robustness of the trained models, practitioners may want to evaluate the model’s performance for different camera distances to see where data gaps are present in the model, or to inform their field acquisition strategies. In such a scenario, using a PBGM would prove very useful, as images could be acquired with different camera paths, and the accuracy of predictions of a fixed trained model can be studied. In this experiment, we train two different ResNet 45 models to predict component damage states. One model is using only the QuakeCity training dataset and is tested on images from another building. Two test sets are prepared, one simulating a UAV camera for data acquisition (UAV-B12), and another simulating a person on the ground collecting images of the structure by pointing the camera forward and upward (Ground-B12). Together, the datasets are referred to as B12. Another model is trained with the QuakeCity training dataset plus 25% of the images from B12 (QuakeCity + 0.25 B12) and evaluated on 75% of the B12 data (0.75 B12). The results of performance on the ground data are reported separately for the UAV and Ground parts of B12. An important aspect of generating effective PBGMs is the selection of various parameters in the five different steps of the proposed framework. The choice of these parameters will most certainly have an influence on the realism of the resulting simulations. In the absence of any data about the structure’s properties, several arbitrary assumptions were made in relation to the structure geometry, material properties, paint color, ground shaking intensity, etc., based on published literature. However, for autonomous inspection studies, the key requirement is the ability to generate large amounts of diverse data, and thus such assumptions, while not perfectly realistic, were reasonable for this research. The proposed framework for data generation has several components that contribute to the relatively high computational cost of generating the data. Running non-linear time history analysis for each component of the structure is very computationally expensive. Leveraging the regularity of the plan in the buildings, and the fact that the same material model will produce similar responses at different floors for identically shaped components, the analysis was run for just one story with different intensities and re-purposed for use with other stories. This simplification greatly reduced the overall computational time. The 3D synthetic environment for one damaged building can be created in about 8 h of total time on a PC with 2 Nvidia RTX 2080 Ti, Intel i7-8700K, and 16 GB of RAM. The rendering of each image then takes about 1.5 min, and each annotation takes about 0.05 min using Blender Cycles. The distribution of time taken for each component of the framework is provided in The experiments conducted demonstrate the efficacy of the proposed framework as a testbed for end-to-end validation of autonomous inspections. The first experiment involved implementing the proposed framework to generate the QuakeCity dataset. The quantity and diversity of data generated in the QuakeCity dataset underscore the benefits of using 3D synthetic environments to generate data to study algorithms for autonomous inspections. While such studies were out of the scope of this manuscript, the dataset has been released as part of the International Competition on Structural Health Monitoring; over 150 teams of researchers are participating to study the performance of different algorithms with the dataset. The second experiment was conducted to study the utility of synthetic data generated from the proposed framework to directly augment deep networks trained for inference on real data for autonomous inspections. The results demonstrated that the use of synthetic data allowed the deep networks to learn better features that resulted in better performance on real data. The transferability of features learned on synthetic data to real data makes the use of the synthetic environment even more attractive. The third experiment illustrates another use case of the proposed framework to study the ability of already trained networks to perform on new scenarios. In the experiment, a deep network trained on UAV acquired data for physics-based damage state estimation is applied to data collected from the ground. The poor results, in this case, indicate that additional data would be required from a ground viewpoint to have an effective network. The addition of about 25% of the data from a single survey was found to increase significantly the performance of the network. Given the cost associated with acquiring data in the real world, such studies are crucial in efficiently developing inspection systems for use in field applications. This paper proposed a framework for generating physics-based graphics models (PBGMs) as part of a 3D synthetic environment that can support the development of automated inspection strategies for civil infrastructure. The proposed framework involved combining the response of a non-linear finite element model to inform the realistic visual rendering of different damage types. The framework was implemented for eleven reinforced concrete building structures subject to earthquake excitation and the damage types rendered included cracks, spalling, and exposed rebar. Three applications were demonstrated for the proposed framework. First, images were rendered from the damaged structures, pixel-level ground truth was generated for the various damage types, for components, component damage states, and depths. The QuakeCity dataset will serve as a benchmark dataset to study the use of deep learning algorithms in automated post-earthquake inspections of building structures. Second, the efficacy of the proposed framework in generating synthetic data to augment real data was demonstrated. It was shown that the performance of models trained with synthetic data and real data performed up to 10 IoU points better than models trained with only real data. Finally, a third experiment was conducted comparing the performance of trained models on the ground and UAV-based data. The experiment demonstrated the utility of the proposed framework for studying and quantifying the value of additional information for models trained for visual inspections. The results demonstrate the immense potential of using PBGMs as an end-to-end tool for the development and study of visual inspection systems.


The feeding region, the mixing process, and the particles settling time could be detected from USCT data. Reactive crystallization experiments for CO2 capture were then conducted. Moreover, there was further potential for quantitative characterization of the suspension density in this process. USCT-based reconstructions were investigated for several experimental scenarios and operating conditions. This study demonstrates a real-time monitoring and fault detection application of USCT for reactive crystallization processes. As a robust noninvasive and nonintrusive tool, real-time signal analysis and reconstruction can be beneficial in the development of monitoring and control systems with real-world applications for crystallization processes. A diverse range of experimental studies shown here demonstrate the versatility of the USCT system in process application, hoping to unlock the commercial and industrial utility of the USCT devices.In this work, an ultrasound computed tomography (USCT) system was employed to investigate the fast-kinetic reactive crystallization process of calcium carbonate. USCT measurements and reconstruction provided key insights into the bulk particle distribution inside the stirred tank reactor and could be used to estimate the settling rate and settling time of the particles. To establish the utility of the USCT system for dynamical crystallization processes, first, the experimental imaging tasks were carried out with the stirred solid beads, as well as the feeding and stirring of the CaCO Industrial Process Tomography (IPT) offers great prospects to visualize the state of vessels, pipes, and tanks containing multi-phase flows and mixtures without being destructive and invasive to the process. Nowadays, there is a great need for monitoring the internal functions of the processes to further improve the design and operation. Therefore, the application of IPT in a wide variety of industrial applications is thoroughly being researched, such as fluid transport in pipes, and the manufacturing of paint, detergents, food, cosmetics, and pharmaceutical operations.Industrial reactors are broadly used in different unit operations, making the tomographic studies on their functionality a hot topic for the research communities ,4,5,6,7.Crystallization is a key reaction in many pharmaceutical/chemical sectors. Among several types of crystallization processes, the industrial demand for reactive crystallization  has been increasing in recent years . The groContrary to the abovementioned crystallization monitoring single-point measurement methods, there has not yet been any applicability of tomographic methods for the control of such processes. Tomography could add useful information regarding the topology and the material phase distributions in mixtures. It can also offer the opportunity to quantify the degree of homogeneity of particulate suspensions and other multiphase mixtures. Even though tomographic techniques cannot compete against the spectroscopic methods in PSD measurements, they can add useful spatial information and can be used as a fault detection or quality assurance tool for the complex mixing processes. They can also be used as complementary alternative measurements to the PSD single-point measurements aiming at multiple measurements integration. Few tomographic studies have been conducted for monitoring in situ crystallization processes such as X-ray microtomography ,26, X-raIn the present work, an ultrasound computed tomography system was utilized to investigate the reactive crystallization process of calcium carbonate. USCT offers some major advantages over other tomographic methods, offering a low-cost alternative that could reach suitable outcomes in spatial and temporal resolution. The calcium carbonate production is integrated with a carbon capture process where chemical . The utiIt is worth noticing that the USCT technology has reached remarkable image resolution that is comparable to MRI and XCT for various medical applications, as well as geophysical applications ,35,36. TThe USCT system in this study is based on transmission-mode USCT data. Recorded waveforms are used to reconstruct travel-time delays and acoustic attenuation (AA) profiles using the time-of-flight (TOF) of the first arrival pulse and its amplitude, respectively. The USCT hardware system consists of a circular ring of 16 transducers, an electronic hardware setup for emitting and recording TOF and AA data of the first arrival pulse, and computer software for analysis and reconstructions. Each recording comprises 256 measurements, accounting for 16 recordings for every one of the 16 emissions that take place. The system provides 5 frames of computed TOF/AA values each second, hence resulting in an overall temporal resolution of 4 frames per second (fps). Such a response over time was used for tomographic reconstruction execution in the present work and is considered adequate for a wide range of dynamical processes, such as pipe-flow monitoring. The USCT system’s design features are presented in The system consists of eight four-channel measurement cards connected by an FD CAN bus with a measuring module, which is a bridge between the microprocessor measuring system and the panel. The measuring module monitors the course of the measurement, stores parameters, controls the high-voltage converter, and switches the bus. The four-channel measurement card was made in a modular manner. The sections of amplifying and filtering analog signals were made on small separate PCBs. For better isolation of individual channels, it is possible to mount a shielding housing on each module. In addition, a four-channel high-rectangular-voltage generator was designed at the bottom of the plate. The generator circuit consists of four four-channel MOSFET drivers connected to double H bridges (TC8220). A single section can generate a three-stage square wave signal (Vpp–GND–Vnn). Depending on the power source, the circuit can generate voltages from +100 V to −100 V. In the presented prototype, the generator circuit is powered by a symmetrical voltage of +/−24 V. Therefore, the maximum instantaneous current efficiency of MOSFET keys is 3 A. The converters were synchronized with a built-in counter in the STM32G474 microcontroller. The sampling data travel to the internal memory buffer by DMA. The generation of appropriate waveforms controlling the MOSFET keys of the four-channel high-voltage generator was carried out by cascading three meters. The microcontroller generates 4 square waveforms . The control signals travel directly to all four generator sections. Each section has an input that activates or deactivates a given section. Thanks to that, it is not necessary to generate control signals separately for each section. The analog module is a system that amplifies the ultrasonic transmitting signal. The proposed design has an integrated AD8331 amplifier with gain control using an external DAC converter and a system converting the AC signal to ADL5511 envelopes and two THS4521 differential amplifiers. Due to the distance between the modules and the microcontroller and the presence of a high-voltage generator on the same PCB, the final output signal from the module is a symmetrical differential signal, which reduces the amount of noise.x-axis (time step axis), shown by the red point in the corresponding graph. The biggest y-value  within a 20% signal’s window in the transmitted pulse “region” indicates the recorded pulse’s amplitude, depicted by the black point in the graph. A more detailed method for signal picking can be seen in [ seen in , where t seen in . The MCU seen in ,40.TOF data and Amplitude Attenuation (AA) data. TOF measurement data come from the subtraction of background data from the full data, and they define the travel-time delays in microseconds (μs).AA measurement data [The tomographic approach uses the transmission sensitivity matrix that simulates the propagation of the measured energy from sensors. The measurement data for transmission tomography include ent data  are compTOF or AA values, one can calculate the average sound speed or acoustic attenuation for the path of the modeled ray. To generate the propagation model of the emitting energy, a computational model based on diffraction on the 1st Fresnel zone is utilized [r), respectively, and Each spherical wave can be approximated by a cone of rays and, subsequently, every plane wave by a fan of rays. Using the recorded utilized . Fresnelutilized . Using aD(x) is the sensitivity at domain’s point x, V is the amplitude at x of the wavefield propagating from source s, V is the amplitude at receiver point r with the wavefront emitted from point x, and K is the normalization constant. The cosine factor models the alternating sensitivity being positive in the odd Fresnel zones and negative in the even Fresnel zones. TOF and AA.The sensitivity of a Frechet kernel based on the first Fresnel zone is defined as follows: kernels . To deve kernels . The senA generalized tomographic forward problem uses the approximated excitation’s sensitivity model and the known domain to recover the measurement data. In our approach, TOF or AA differences can be used as measurement data p:Generally, the LBP leads to low-quality USCT imaging results. Advanced regularization-based methods can produce high-quality images. To solve the regularized inverse problem with a high degree of stability, the robust total variation (TV) regularization algorithm is applied . The TV lgorithm . To optiUltrasonic measurements have a proven efficiency in the characterization of suspension densities and slurry mixtures. Prior studies have shown the relation between phase velocity and acoustic attenuation on growing suspensions and different frequency excitations ,47. Thes2-loaded solutions, and later using them for calcium carbonate crystallization. In the process under investigation, the semi-batch feed to the stirred tank reactor contained dissociated 2 capture process and its integration with the calcium carbonate crystallization is given in [The micron-sized crystallization process of the present work is part of the carbon capture and utilization scheme where process monitoring with the USCT system is presented. Carbon dioxide is scarcely soluble in water under the standard temperature and pressure conditions. given in .(11)CO3The feed addition rate to the receiving reactor was constant at 40 Several dynamical experiments were specifically designed to evaluate the USCT response to various process events. The effects of certain operator-induced malfunctions including switching on/off the stirrer and the feed pump, particle addition, and phase changes were investigated. A description of the experimental results is presented in the following sections. In the presented work, we show the potential applications of the USCT in chemical reactor processes and especially in batch crystallization. USCT characterizes the medium by measuring the time of flight and acoustic attenuation of the sound propagation. Ultrasound tomographic imaging offers promising industrial applications due to its low-cost value and the ability to be nonintrusive to the whole process even though it is considered invasive to the process. First, the experimental tests focused on particles’ concentration and dynamical status characterization in mixing scenarios. As mixing highly affects the dynamical state of the tanks, it introduces noise to the system, which disturbs the ultrasonic measurements. Noise is always apparent in the measurements when the dynamics of the tank abruptly change. The main challenge, regarding ultrasonic online measurements in mixing environments, is the elimination of the “stirring noise” and the use of the quantitative information, without the measured data being seriously distorted.To establish the dynamical imaging based on particle concentration, circular particle beads of 4 mm in diameter were poured into the stirred tank reactor containing 3 L of water. The idea of the proposed experiment was to investigate the effects of mixing and the real-time changes in the dynamical states of the reactor by using USCT. A description of the experimental procedures is presented in As tabulated in The exact experimental procedures as discussed above  were conThe experiment with the plastic beads offers insights into the system’s functionality in high-dynamical scenarios. First, efficiency was proven in detecting malfunction cases coming from localized higher concentrations and generated disturbances in the medium (by pouring). Then, quantification was achieved in different concentration tests. Finally, the specific flat blade turbine impeller introduced the specific pattern of gathering the plastic beads close to the stirring area (tank’s center).Four different concentrations of solid calcium carbonate suspensions were used to investigate the sound propagation. Samples of commercial calcite  were added by hand from the top of the reactor containing 3 L of water. 3 particles. Switching on the mixer at T + 5 min distributes the micron-sized particles in the suspension and causes a rapid increase in the TOF. The increase in the TOF delays agrees with the concentration increase. The TOF has a descending trend after reaching the maximum peak (ca. from T + 5.5 min afterward), which characterizes a relatively homogenous medium that facilitates sound propagation, concluding in the sound speed propagation decrease at higher concentrations of solid calcium carbonate in the reactor, resulting in higher TOF delays. For the corresponding experiments, AA data were analyzed, offering the same responses as the TOF data.As presented in , is determined based on the following expression:g is the gravitational acceleration : it decreases as the solid particle concentration increases in the fluid domain. The free settling velocity of particles for the Stokes’ regime, In the present work, the settling period was approximated based on the ultrasound tomographic measurements of the most diluted suspension  and (13) for the abovementioned particle size, the average settling period is approximately 1 min for the solid volumetric concentration of 0.31% . In all the cases, the initial concentration of calcium chloride was 1.6 Two instances of the mean value of sound speed are presented in Nucleation determines the main properties of the crystal population, including the crystal polymorph, the number of crystals, and their size distribution. In the current precipitation system, there is a possibility for the following succession of mechanisms to occur ,51: (i) Furthermore, ultrasound tomographic reconstructions provide deeper insight into the precipitation process of calcium carbonate. Data of Trial 1 in As demonstrated in the experiments with particle beads , USCT meIn reactive crystallization, the time-of-flight USCT provides useful information on feeding points and the later stage of the material phase change and crystal growth. Referring to the mean value plot of TOF data in To conclude, the reactive crystallization experiment was a full-scale test on live-process challenges. In the specific process, the system showed capability in detecting the injection point, as a prior localized suspension in the form of ACC. Moreover, providing real-time measurements of travel-time delays’ mean values aims at the inhomogeneities’ characterization over time and subsequently the dispersion state of the mixture. Finally, acoustic attenuation data proved to be more sensitive compared to the time-of-flight responses, as their response was significantly better in the malfunction analysis. Concluding, the mean value data for each experiment of USCT were plotted to show a global picture of the process’ progress and dynamics in all cases. Distributed sensing measurement systems such as process tomography provide the capability to measure spatiotemporal field information from within the process. Therefore, the reconstructed images aided in the injection point and in the mixture’s homogeneity characterization. The USCT images showed the feeding process, the mixing process, and material phase changes during the crystallization process. Moreover, USCT data were further analyzed to estimate the settling velocity and settling time of particles in the suspension.2 capturing via reactive crystallization is becoming a new tool to reduce the negative environmental impact of CO2, helping in net-zero targets. There is clear indication that the USCT tool can be used to monitor such a process noninvasively and could then become a practical tool to process design that can maximize the overall process yield contributing to affordable carbon capture and carbon reduction. To develop a complete understanding of a complex industrial process, such as carbon capture experiments, it is likely that a multi-modality sensing and imaging approach will be required. In such a multimodality imaging setup, the data-rich USCT is likely to play a vital role. The work of this paper demonstrates that USCT is a very attractive and proven tool to many industrial applications. Hopefully, this work will stimulate further commercial exploitation of the USCT technology.Transmission-mode USCT was examined for industrial process monitoring. The experiments were designed in such a way that enables critical evaluation of the USCT and its usability for process monitoring and potentially for process control. For transmission USCT, we implemented both time-of-flight, allowing speed-of-sound imaging, and amplitude attenuation imaging. Although we mostly showed the time-of-flight data, the amplitude attenuation was also examined and provided similar or, in some cases, complementary information on the state of the process. The major events on the process, such as feeding, and switching on and off the pumps and the stirrer, could be seen in the data. These indicators could be integrated into the process and be used for process malfunction. CO


Background: The goal of the study was to create a histopathology image classification automation system that could identify odontogenic keratocysts in hematoxylin and eosin-stained jaw cyst sections. Methods: From 54 odontogenic keratocysts, 23 dentigerous cysts, and 20 radicular cysts, about 2657 microscopic pictures with 400× magnification were obtained. The images were annotated by a pathologist and categorized into epithelium, cystic lumen, and stroma of keratocysts and non-keratocysts. Preprocessing was performed in two steps; the first is data augmentation, as the Deep Learning techniques (DLT) improve their performance with increased data size. Secondly, the epithelial region was selected as the region of interest. Results: Four experiments were conducted using the DLT. In the first, a pre-trained VGG16 was employed to classify after-image augmentation. In the second, DenseNet-169 was implemented for image classification on the augmented images. In the third, DenseNet-169 was trained on the two-step preprocessed images. In the last experiment, two and three results were averaged to obtain an accuracy of 93% on OKC and non-OKC images. Conclusions: The proposed algorithm may fit into the automation system of OKC and non-OKC diagnosis. Utmost care was taken in the manual process of image acquisition . Further, there is scope to improve the accuracy rate and make it human bias free by using a whole slide imaging scanner for image acquisition from slides. Artificial Intelligence and machine learning has evoked interest and opportunities propagating research in health care. The newly developed automated tools that target varied aspects of medical/dental practice have provided a new dimension to translate the laboratory findings into clinical settings . The autHowever, if one focuses on the source and output, mere automation of images may not yield the desired results. Thereby, it is necessary to maintain the patient clinical details and follow-up data for a minimum of five years .Pathology is the branch of medicine that deals with the microscopic examination of biopsied tissues for diagnostic purposes. The clinical diagnosis done by medical professionals mandates pathologist’s consultations to tailor the treatment . Thus, hA routine non-digitalized diagnostic pathology workflow involves procurement, preservation, processing, sectioning, and staining of the biopsied tissue to create glass slides followed by an interpretation . The chaThe major Machine Learning challenges to analyze the histopathology images include 1) The requirement of a large dataset to analyze the histopathology images through machine learning algorithms. (2) Identification and assessment of biological structures such as nuclei, with varied shapes and sizes. (3) To detect, analyze, and segment tissue structures in the stroma, such as glands and tumor nests. 4) Lastly, to classify the entire slide image with stroma and epithelial cells  is obtained by concatenating the feature maps from layers 0 to n − 1. Cn is a composite function of three operations in the order: Batch normalization, ReLU, and 3 × 3 convolution.[The architecture of the Dense Net is divided into several blocks referred to as ‘dense blocks. These blocks are separated by transition layers, which consist of convolution and pooling layers. This CNN captures the overall image features including the unique feature of OKC, separation between epithelium, and connective tissue region. This also helps in capturing the inflamed OKC, where the tombstone arrangement of basal cells was disturbed.DenseNet169 was trained on the preprocessed dataset. This dataset was created by retaining the patches with epithelium layer in OKC and non-OKC (DC and RC) as explained in the preprocessing section. This CNN is trained to capture the regularity in basal cell arrangement (tombstone arrangement) and 5 to 8 layers of basal cell.The models trained in experiments II and III were integrated by averaging the resultant confidence scores to get the predicted output.The overall architecture of experiment IV is as shown in The optimizer used is the Adam optimizer, and binary cross entropy is the loss function.There were 1384 images of OKC and 1273 images of non-OKC; 15% of the images were used for testing the trained model. In total 207 images of OKC and 191 images of non-OKC were used for testing.The loss function used in each classifier model discussed here is binary cross entropy. This is most used for binary classification problems. The formula for calculating the binary cross entropy loss or log loss is given by Equation (2).Accuracy is the fraction of several predictions done correctly by the model out of the total number of samples. The formula for calculating accuracy is given by Equation (3).Precision is the fraction of several true positive cases out of the number of samples that are predicted positively by the model. The formula for calculating precision is given by Equation (4).Recall is the fraction of many true positive cases out of the number of actual positive cases. The formula for calculating recall is given by Equation (5).The F1-score is the harmonic mean of precision and recall. The formula for calculating the F1-score is given by Equation (6).ROC or Receiver Operator Characteristic curve is a graph that plots the true positive rate against the false-positive rates at different threshold values. This is particularly useful in binary classification problems.A confusion matrix is a table that gives us a summary of the model’s performance. The format of the confusion matrix for a binary classification problem is shown in .The macro-average gives the overall performance of the classifier. The macro average is the arithmetic mean of individual classes’ precision, recall, and F1-score.Weighted avg gives the function to compute precision, recall, and F1-score for each label and returns the average considering each label’s proportion in the dataset.In the training phase the parameters given to the developed model were: (a)‘Adam’ optimizerAn optimization algorithm plays an important role in deep learning algorithms, as it is a strategy that is performed iteratively until an optimum solution is obtained. Adam optimizer is a hybrid of Adagrad and RMSProp algorithms to produce an optimum solution for a given problem.(b)minimum batch sizeUpdating the internal model parameters would be tedious if done after every sample, so samples are grouped as batches and the model parameter is updated for these batches.Batch size is a hyperparameter. Here, 11 histopathological images were grouped as batches.(c)the number of training epochsOne epoch means the entire training dataset was used to update the internal model parameters once. The number of epochs is a hyperparameter.(d)initial learning rateThe learning rate is also a hyperparameter. In the case of stochastic gradient descent optimization algorithm learning rate is the amount of the internal model parameter to change concerning the calculated error.Hyperparameters such as the number of epochs and batch size are decided based on experimentation and comparing the results.In experiment I the VGG16 model performed its best when trained for nine epochs with 12 as the batch size. At the end of nine epochs, the validation accuracy is 89.01% and test accuracy is 62% as shown in In experiment II, only step 1 data augmentation was performed in preprocessing. That is, the CNN was trained only on the entire image.In experiment II, the DenseNet169 model performed its best when trained for 15 epochs with 12 as the batch size. At the end of 15 epochs, the validation accuracy is 89.82% and test accuracy is 91%. The AUC for this model is 95.966%.The plot of accuracy on training and validation data is as shown in In experiment III, both the steps were performed in preprocessing. That is, the CNN was trained only on the features of the epithelium.On experimentation and comparison, Model-3 gave its best predictions when trained for 100 epochs with 16 as the batch size. At the end of 100 epochs, the test accuracy is noted as 91%. The AUC for this model is 96.375%.The plot of accuracy on training and validation data is as shown in B. From tThe plot of loss on training and validation data is as shown in B. From tIn experiment IV, the models trained in experiments II and III were integrated by averaging the resultant confidence scores to get the predicted output. The confusion matrix and classification report are shown in The main features to look at in the OKC histopathology image pattern are thick epithelium, 5–6 layers of a regular arrangement of basal cells, also called tombstone arrangement, and the separation of epithelium and the connective tissue. On the other hand, DC has a thin epithelium layer with 2–3 layers of irregularly arranged basal cells and the epithelium layer is not properly distinguishable and has penetration into connective tissue in RC. The classification is more challenging when they undergo inflammation, the epithelium layer of DC becomes thick, which may be confused for OKC.These images show large diversity within the class, posing a challenge for automation. Deep learning has shown promising results in the automation of digital histopathological image classification. Automation of digital histopathological image classification brings the standardization in the procedure, by eliminating the manual observation of the tissues under a microscope by the pathologists, which is completely subjective and suffers inter-laboratory variations.In the proposed design, during preprocessing an attempt was made to select the epithelial region in the images. A Dense Net CNN was trained on these datasets to identify the features in the epithelium which distinguishes OKC from RC and DC. Another Dense Net was trained on the whole image to capture the separation of epithelium and the connective tissue feature. As the dataset had a limited number of images, transfer learning was adopted. The transfer learning approach results in better accuracy and reduced training time. Transfer learning is reusing a pre-built and pre-trained model on a new dataset. With transfer learning, we can reduce training time and get the best results even when many of data are unavailable. Both Dense Net CNNs obtained 91% accuracy over the dataset. An ensemble model of the Dense Net CNNs obtained an accuracy of 93%. Densenet169 was pre-trained on the ImageNet dataset only the last layer was trained for the given dataset. The problem of the OKC and non-OKC classification was addressed by using two CNN models. First, CNN was trained on the hand-crafted small patches of epithelium and the second on the whole image. They obtained a higher accuracy of 98% as they had a large dataset of 1704 OKC and 1635 non-OKC images .The classification of periapical cysts (PCs), dentigerous cysts (DCs), ameloblastomas (ABs), odontogenic keratocysts (OKCs), and normal jaws with no diseases, considering the dataset for panoramic radiographs, were also attempted . The claThe current investigation used H&E-stained sections on incisional biopsies, which are globally acknowledged, cost-effective, and time-tested. However, there are other ways to reduce the dataset by opting for immunohistochemical (IHC) staining specific to the nucleus. In the absence of a typical epithelium, in case of inadequate biopsy, the presence of non-keratinizing epithelium with basal palisading and an immunophenotype characteristic of OKC  may be consistent with the OKC diagnosis . A basicThe hematoxylin and eosin-stained tissue specimens of OKC and NK were collected as a dataset. Two convolutional neural network models were trained on the region-selected dataset and the whole image dataset separately. These were ensemble by averaging their confidence scores, to give better accuracy. This architecture could be computationally expensive and may require a faster CPU to overcome. If IHC is opted for as a choice to reduce the dataset, utmost care must be taken on the economic viability while using IHC.