Acoustic black holes (ABHs) are effective at suppressing vibrations at high frequencies, but their performance at low frequencies is limited. This paper aims to improve the low-frequency performance of ABH plates through the design of a metamaterial acoustic black hole (MMABH) plate. The MMABH plate consists of a double-layer ABH plate with a set of periodic local resonators installed between the layers. The resonators are tuned to the low-frequency peak points of the ABH plate, which are identified using finite element analysis. To dissipate vibration energy, the beams of the resonators are covered with damping layers. A modal analysis of the MMABH plate is performed, confirming its damping effect over a wide frequency band, especially at low frequencies. The acoustic black hole (ABH) is an acoustic analogy of the concept of a black hole in astrophysics within the realm of mechanics. In 1946, Pekeris  discoverIn recent years, there have been numerous applications based on ABH, and research in this field has expanded into four areas: vibration reduction, wave regulation, noise radiation, and energy recovery. Lagny et al.  studied The ABH plate has several advantages over other methods in terms of vibration reduction and sound radiation, but also has some limitations. One of the limitations is that it only operates above the cut-on frequency fcut−on . This mefcut−on .Few methods have been proposed to improve the low-frequency performance of ABH to date. In 2014, Feurtado investigated the reflection of flexural waves via ABHs when introduced to uniform beams without anechoic terminations through numerical simulations . FoucaudThis paper presents two main contributions. Firstly, it proposes a double-layered ABH plate consisting of two identical ABH plates with circular indentations, connected by supporting bars. The design enhances the structure’s rigidity and allows for multimodal local resonators to be mounted on the supporting bars. Secondly, the paper presents a real multimodal resonator design that can suppress multiple low-frequency peaks. Unlike previous theoretical and single-modal designs, this resonator is based on a cantilever beam-type local oscillator, with four oscillators mounted on a single supporting bar to achieve a multimodal design. The resonators are tuned to form band gaps that suppress low-frequency vibration peaks in the double-layer ABH plate, while their beams are damped to reduce other vibration levels in the lower frequency bands. The interaction of the multimodal local resonator with the double-layer ABH plate is investigated in this work. Importantly, the mass of the ABH indentation that is removed from the uniform plate is recycled as the mass of the resonators, ensuring that the MMABH design does not increase the weight of the original uniform plate.The organization of the paper is as follows. In this section, the MMABH structure is described. At the outset, a high-stiffness double-layer ABH plate was proposed and its mean square velocity (MSV) was analysed when subjected to external forces. Peaks in the lower-frequency bands were observed, based on which multimodal local resonators were designed and tuned to exhibit band gaps at these peaks. In addition, an overlay was applied to dampen the resonators, significantly reducing the peaks at the remaining lower-order eigenfrequencies. Finally, the design of the MMABH plate was completed by integrating the multimodal local resonators with the supporting bars of the double-layer ABH plate, resulting in broadband damping that covers the entire frequency range, especially at low frequencies. This approach resulted in a highly effective solution for mitigating vibration-induced noise in ABH plates.Geometric and material parameters for a single ABH plate configuration are provided in tiffness  and is ng effect , the sidFinally, the purpose of designing this structure is presented. The double-layer ABH plate designed in this study has a black hole radius of 0.15 mm, which provides a suitable shear range and better load-bearing performance. Moreover, the supporting bars offer a convenient location for mounting the resonators.After introducing the structure of the double-layer ABH plate, it was modelled using the finite element software COMSOL Multiphysics 6.0. The simulation was kept simple by setting simple support boundary conditions around the boundaries of both upper and lower plates and applying external forces to observe the structure’s response. The force was assumed to have a constant amplitude of 2 N for all frequencies, directed vertically towards the plate, and located at  m. Its mean squared speed was then calculated .Firstly, we simulated the modal frequencies of the MMABH see . As showf > 548 Hz. Nevertheless, the MLFs of the ABH plate are very small in the low-frequency band. The MMABH solves this problem by adding multimodal resonators to the double-layer ABH plate, resulting in a very low amplitude and slight vibration energy loss at these frequencies. When overlaid damping is applied, the MLFs are significantly increased. As shown in Secondly, Finally, taking the mode shape diagram of D-MMABH in Group a as an example, for the first order  m. The MSV figure of ABH-MMABH see  can be dTo gain further insights into the behaviour of the MMABH plate, we calculated and plotted some of the forced vibration shapes in Regarding the activation of resonators, the two resonators below the connecting rod are activated at a vibration frequency of 197 Hz, the resonator in the upper left corner is activated at a vibration frequency of 234 Hz, and the resonator in the upper right corner is activated at a frequency of 265 Hz. The slight displacement of the connecting rod at 376 Hz is also noteworthy. These results indicate that the resonators in the MMABH plate behave as expected.We now investigate the coupling action of the multimodal local resonators designed with two layers of ABH plates. However, it is worth noting that the MSV curve of the MMABH exhibits two troughs in the high-frequency region when the excitation frequency exceeds 548.1 Hz see . To explTo further investigate the effects of the multimodal local resonators, we assembled MMABH plates with damped multimodal local resonators for observation. We provided MSV graphs at different frequencies for the two plates for analysis see . By compIn the high-frequency range, the damping layer has a strong influence on the local modalities, leading to an increase in mass and stiffness . PreviouIn this paper, the design of an MMABH plate capable of suppressing bending vibrations on the broadband spectrum has been presented. It consists of a double-layer ABH plate plus resonators. The resonators are tuned to the three low-frequency peaks of the double-layer ABH plate to create band gaps, while the damping is superimposed to reduce vibration levels at other frequencies. MMABH plates combine the properties of the high-frequency ABH effect with the formation of low-frequency band gaps, which are a promising locally resonant acoustic metamaterial.A modal analysis of the MMABH plate was first carried out to investigate its characteristics, including modal frequency, modal damping, and modal shape. The results show that the characteristic frequencies of the MMABH plate form a stepped image and modal loss factors of the ABH and MMABH plates were almost equal at high frequencies. The broadband damping performance of the MMABH plate under external forces was then investigated, and the results showed that the MSV of the MMABH plate and D-MMABH was lower than that of the uniform and conventional ABH plate.The next task was to analyse the coupling effect between the multimodal local resonator and the ABH plate at different frequencies using the MSV curves and forced mode diagrams of the MMABH. The results show that the resonator is damped at low frequencies but not at high frequencies. In addition, the effect of the multimodal local resonator is further investigated in this paper by analysing the MSV curves of the ABH and MMABH plates under forced vibration. It is found that the resonator has a damping effect at low frequencies and that the damping of the beam flattens the amplitude of the MMABH in the frequency band in which the resonator operates.Overall, the D-MMABH designed in this study exhibits good broadband-damping effects and load-bearing performance. It is a promising metamaterial for local resonance acoustics.


The growing interest in augmented reality applications has led to an in-depth look at the performance of head-mounted displays and their testing in numerous domains. Other devices for augmenting the real world with virtual information are presented less frequently and usually focus on the description of the device rather than on its performance analysis. This is the case of projected augmented reality, which, compared to head-worn AR displays, offers the advantages of being simultaneously accessible by multiple users whilst preserving user awareness of the environment and feeling of immersion. This work provides a general evaluation of a custom-made head-mounted projector for the aid of precision manual tasks through an experimental protocol designed for investigating spatial and temporal registration and their combination. The results of the tests show that the accuracy ( In recent years, the potential of Augmented Reality (AR) technology to enhance the real world with auxiliary virtual information has become increasingly prominent. To this end, Head Mounted Displays (HMDs) are by far the most studied devices. Their application has been proposed in various fields ranging from industrial quality control and assembly ,2, educaProjected AR, also called spatial AR, lends itself to preserving the user’s awareness of the environment and the feeling of immersion since it does not require any optical interface interposed between the user’s eyes and the surrounding world. In projected AR, physical and digital information are mixed directly into the real world via visible light projectors. As a result, a single projected AR device can be simultaneously experienced by multiple users. However, such devices are mostly static, as they are installed in a fixed position, hence they inevitably constrain the user’s freedom of movement. With the development of increasingly smaller and lighter projectors, new solutions have been proposed to overcome this problem. Nazarova et al., for instance, mounted a tracking and projection module on a collaborative robot to provide immersive and intuitive robot control to the user . HartmanThe term MTP latency was originally used to denote the average time between the instant when a motion occurs and the instant when the corresponding rendering event gets displayed on the display . MTP latency has been extensively studied for HMDs and literature results reveal their major impact on the system’s usability  since hiMeasure the temporal registration error of the HMP in terms of MTP latency.Measure the static spatial registration error in terms of accuracy by evaluating the distance between the actual position of the virtual content and the desired position for perfect alignment in static conditions.Measure how these two errors combine when a user wears the device and uses it to augment a tracked structure in real time.In this work, we propose an HMP built by assembling commercial devices, and we provide a detailed description of the assessment of its spatial and temporal registration together with their influence on the usability of the worn system . The spaAll measurements were taken within the working area accessible to a user who intends to perform manual tasks while wearing the HMP. This area is between 350 mm and 600 mm from the user’s head, with an average value of 475 mm .First, we present the hardware specifics of the HMP together with a brief explanation of the software framework that implements the tracking and the rendering. Then, the experimental protocol employed to evaluate the spatial and temporal registration accuracy and the combined effect of MTP latency and virtual-to-real static registration is described during a user test.The custom-made HMP includes a Leopard Imaging OV580 stereo camera rig and an Anybeam laser mini-projector. The adopted camera configuration is 2560 × 720 @ 60 fps, which corresponds to an angular resolution of about 3.6 arcmin/px. The AnyBeam projector offers the advantage of no need for focus adjustment, as this is ensured by the MEMS laser technology. Projector resolution and brightness are, respectively, 720p @ 60 fps and 150 lumens. The stereo rig and the projector are calibrated to estimate both their intrinsic and extrinsic parameters ,22. The The software, originally conceived for the deployment of AR applications on video see-through (VST) and optical see-through (OST) display interfaces, is capable also of supporting the deployment of AR applications on different typologies of AR projectors. The software features a non-distributed architecture, which makes it also compatible with embedded computing units.The software framework is based on Compute Unified Device Architecture (CUDA by Nvidia) to leverage parallel computing over the multi-core GPU card.The software supports in situ visualization of 3D structures, thanks to the employment of the open-source library VTK for 3D computer graphics, modeling, and volume rendering.The software framework is highly configurable in terms of rendering and tracking capabilities.The software features a robust inside-out video-based tracking algorithm based on OpenCV API 3.4.1,The software framework that controls the HMP was thoroughly described in . Here, wThe software framework is able to support in situ visualization of medical imaging data; the key function of the software, under projector AR modality, is to process the images grabbed by the stereo pair of RGB cameras. The grabbed frames of the real scene are processed to perform marker-based optical tracking, which requires the identification of the 3D position of the markers both in the target reference frame and camera reference frame. The projected AR device needs a dedicated calibration between the world-facing tracking cameras and the projector. For this purpose, the projector is considered as an inverse camera according to the procedure exposed in . Such caThe experimental protocol relies on an external device for the evaluation of spatial and temporal registration accuracy. We selected a GoPro Hero 5 Black camera that features either a 4000 × 3000 photo resolution or a frame rate of up to 240 fps for videos at a 1280 × 720 resolution. As shown in We evaluated the AR spatial registration accuracy as the on-image distance between the location of a real point and its virtual counterpart . For this purpose, we adopted a spatial registration evaluation tool consisting of three spherical markers fixed on a plane featuring two elevated points. The tool was modeled using Creo Parametric 3.0 and printed by rapid prototyping with a 600 dpi X-Y resolution, 1600 dpi Z resolution, and 0.1 mm precision. The two target points were dyed in red to be easily identified in the images as they are the reference points for AR alignment error assessment. The experimental set-up is shown in For the evaluation of MTP latency, we chose to adopt the protocol proposed in . SpecifiTo evaluate how the spatial and temporal registration errors combine whilst the HMP is worn by a user, we adopted the dummy presented in . The dumThe spatial registration error within the working area was 6±0.1 mm . These sFor the temporal registration, the number of frames elapsed between the motion of the evaluation tool and the update of the virtual information is 13±12 ms  This valWe recorded a 25 s clip during the user tests which corresponds to about 6000 frames. Each frame was processed to estimate the centreline of the real and the projected cutting lines. Their Hausdorff distance is plotted in 9±0.2 mm . The regIn terms of perceived user comfort and ergonomics, weight distribution in head-worn displays and projectors is one of the main limiting factors that hinder the widespread adoption of mobile AR devices in the healthcare sector and specifically in image-guided neurosurgery ,31. By wWe proposed a device for projected AR, which can be worn by the user to guide the execution of precision manual tasks while preserving mobility in the working area. Compared to HMDs, our HMP offers the advantages of being simultaneously accessible by multiple users, without altering their awareness of the environment and feeling of immersion. We evaluated the performance of our HMP in terms of spatial and temporal registration along with their mutual interaction in a user test. The latency of 113 ms is a positive result, especially considering that the HMP is not equipped with an IMU sensor, nor with predictive slam algorithms or fast-rendering engines such as the ones adopted in the HoloLens. Specifically, the HMP meets the specifications proposed in  for HMDsTo the best of the authors’ knowledge, our evaluation of spatial and temporal registration, together with their mutual interaction in a user test constitutes the first overall assessment of an HMP for precision manual tasks. We hope that this will provide a reference for further investigation of this and other AR HMPs and HMDs contributing to providing a standard for their performance assessment.


Instrument separation is one of the most upsetting endodontic errors, and it can occur at any time during root canal surgery. A broken file causes a metallic restriction in the root canal and makes it difficult to clean and shape the instrument effectively, thereby making the prognosis questionable. Hence, such a fragment should be properly retrieved when it becomes difficult to bypass it. A variety of techniques and tools are utilized for instrument recovery; however, most of them are expensive, difficult to master, and technique-sensitive. One such tool for orthograde removal of intracanal metallic obstacles is the Masserann kit. We discuss a case involving file removal from the maxillary premolar by using the Masserann technique. Instrument separation is very common in root canals, with fascinating anatomical variations that make the treatment more difficult. These anatomical differences in the canal put patients at risk of a range of iatrogenic issues, including missing canals, instrument separation, gouging, perforation, and excessive extension of the obturation materials. One such unfavorable occurrence is instrument breakage, which may impede the cleaning and shaping processes and cause pain or discomfort . InstrumThere are various methods and tools to retrieve separated instruments, such as manual instrumentation, ultrasonic technology, canal finder systems, Masserann kits, the Endo-Extractor System, and various pliers as well as chemical agents like iodine trichloride. In particular, the dental operating microscope validates the proverb "If you can see it, you can probably do it," by enabling physicians to see most broken devices . The MasA 55-year-old female patient presented to the Department of Conservative Dentistry and Endodontics, Sharad Pawar Dental College and Hospital; she had been referred from the Prosthodontic Department for intentional root canal treatment of the upper right first premolar (tooth number 24) and second premolar (tooth number 25). Access opening was performed with a No. 1 endo access bur in 24 and 25 following the administration of local anesthesia. Both 24 and 25 contained two canals, the buccal and palatal canals. The working length was established and the cleaning and shaping of the root canal procedure were initiated. A radiograph was acquired to confirm the separation of the W3-path (12/.vt) , which was separated in the buccal canal in 24 during cleaning and shaping, as shown in Figure At the intersection of the middle third and apical third of the canal, around 6 mm of the instrument got broken. Because bypassing the fragment was unsuccessful due to the separated file's length of more than 5 mm approximately, the Masserann technique was employed for instrument retrieval. Utilizing diamond points, the coronal access in tooth 24 was improved . The consecutive application of the Peesso reamer straightened the obstruction's radial access . End-cutting trephan with an acceptable diameter was chosen by superimposing it over the obstruction on the radiograph as the size of the obstruction was unknown. To remove the radicular dentin surrounding the broken instrument, the trephan was slowly spun in an anticlockwise direction. Radiographical monitoring was done to see how far the trephan had advanced. A 1.2-mm diameter extractor tube was inserted into the canal to encircle and hold the obstruction once the coronal section of the impediment was released from the dentin. After radiographic confirmation that the fragment was completely encircled by the extractor tube, the obstruction was tightly gripped against the extractor tube's wall by rotating the plunger rod of the tube in a clockwise manner, as shown in Figure The extractor tube was gently twisted back and forth to loosen and remove the obstruction from the canal once tactile gripping of the impediment was confirmed. The separated fragment was successfully retrieved, as shown in Figure Radiographic evidence supported the removal of the blockage from the canal of tooth 24, as shown in Figure After the impediment was removed from tooth 24, a ledge was found apical to the area of the obstruction. A size 10 K file was utilized to navigate the cliff. The pulp space was further investigated, and another canal was found. After canal negotiations, the working length was calculated using radiography and the apex locator  Figure .NiTi files were used to form and clean canals with the crown-down technique ; 2.5% sodium hypochlorite and 2% chlorohexidine were used for irrigation. As an intracanal medication, calcium hydroxide  was employed. Using Gutta percha points , Dia-Proseal sealer , and single cone technique, the canals of tooth 24 were sealed on the subsequent visit. Composite was used to repair the access aperture, as shown in Figure Instrument breakage prevents proper chemomechanical preparation in root canals, which may negatively affect the treatment procedures. The fragment also inhibits the apical end of the root canal. Rotating instruments can break due to a number of reasons, including the operator's lack of experience, the device's speed and torque, curvature, instrument design and technique, and manufacturing flaws . The manThe best course of action when an instrument breaks, according to Machtou and Reit, is to retrieve it . HoweverInstrument recovery using the Masserann kit is successful in the anterior and posterior regions, with success rates of 73% and 44%, respectively . This toAccording to Ruddle, a staging platform can be created by choosing a Gates Glidden drill whose maximum cross-sectional diameter is marginally greater than the instrument being visualized. The Gates Glidden drill's bud is modified by cutting it perpendicular to its long axis at the diameter of its largest cross-section. This staging platform aided in the introduction of ultrasonic instruments for the safe removal of the peripheral dentin surrounding the exposed fragment, which improved the instrument's hold with the extractor and permitted the retrieval of the fragment .In our case, the separated instrument was tightly lodged in the apical third of the maxillary first bicuspid. Since attempts to get around it were unsuccessful, the Masserann method was utilized. The trephan's centering over the piece was made easier by having direct access to it. By carefully cutting the peripheral dentin surrounding the fragment, this assured circumferential release of the coronal end of the fragment. This provided a firm grasp on the fragment and helped retrieve it along the long axis of the root, enabling routine retreatment.To minimize instrument separation and the accompanying stress and anxiety, proactive prevention should be the primary focus. In cases of separation, opting for safe retrieval or bypassing techniques is essential. The Masserann method is a retrieval procedure that demands both time and effective risk management. Its application is best suited for the extraction of firmly lodged metallic obstructions found in premolars and other posterior teeth and identified as separated instruments. However, it should be limited to straight canal segments with ample surrounding dentin, as it is only effective in such areas and necessitates a high level of clinical skill and unobstructed access along a linear path.


With the growth of the world’s population, limited healthcare resources cannot provide adequate nursing services for all people in need. The wheelchair-mounted robotic arm (WMRA) with interactive technology could help to improve users’ self-care ability and relieve nursing stress. However, the users struggle to control the WMRA due to complex operations. To use the WMRA with less burden, this paper proposes an object affordance-based implicit interaction technology using a laser pointer. Firstly, a laser semantic identification algorithm combined with the YOLOv4 and the support vector machine (SVM) is designed to identify laser semantics. Then, an implicit action intention reasoning algorithm, based on the concept of object affordance, is explored to infer users’ intentions and learn their preferences. For the purpose of performing the actions about task intention in the scene, the dynamic movement primitives (DMP) and the finite state mechanism (FSM) are respectively used to generalize the trajectories of actions and reorder the sequence of actions in the template library. In the end, we verified the feasibility of the proposed technology on a WMRA platform. Compared with the previous method, the proposed technology can output the desired intention faster and significantly reduce the user’s limb involvement time (about 85%) in operating the WMRA under the same task. Nowadays, the world’s population is in the stage of growing . The incTo facilitate the operation of robots for users with different levels of physical ability, many human–robot interactions (HRI) interfaces utilizing residual limb abilities have been studied, such as using chin , shouldeIn recent years, researchers have tried to use the concept of “shared attention ” to reduUsers could convey simple intentions to the robot through share attention. However, people’s intentions are complex and robots’ tasks are unstructured, which requires a robot to make deeper inferences about intention from a simple interaction. The concept of object affordance refers to actions that match the physical properties of an object ,21, whicThe proposed technology is composed of two major parts. One is the laser semantics identification model based on the SVM and YOLOv4 fusion algorithm, which mainly identifies the corresponding semantics by detecting the flicker frequency of the laser spot. The other is the household task intentions reasoning model based on the conditional random field (CRF) and Q-learning algorithm. We formalize the concept of object affordance based on the CRF algorithm and reason the user’s intention through the object information in the scene. In addition, through the Q-learning algorithm, the proposed reasoning model can learn user preferences in the long-term historical intention prediction, in which people may have different operation intentions for the same object on different occasions. Based on the above models, our implicit interaction technology can use the laser pointer to achieve “share attention” with the WMRA and reason the user’s intention. In the end, to execute the household tasks and verify the practicability of the proposed method, we embed the DMP algorithm into the laser interaction technology to generalize the action trajectories of the tasks and achieve the logical state transformation of different actions through the finite state machine.The WMRA with laser interaction is designed, and it can identify the laser semantics correctly even if users point to the wrong position due to hand tremors.The model of intention reasoning and execution based on object affordance is proposed so that the WMRA can identify the desired intention faster and perform tasks with less limb involvement from the user.The contributions of this paper are summarized below:The rest of this paper is as follows. Interaction using a laser pointer has been widely studied because of its simple operation and clear instruction. Gualtieri et al. created a grasping assistance system for activities of daily living (ADLs), and its novel user interface and grasping capabilities enable this system to grasp objects automatically when a user points to an object with a laser pointer . Sprute These studies presented some more user-friendly and low-cost interactive methods with a laser pointer and proved their efficacy. There would not be a heavy burden on the human body when equipped with a laser-pointing device, and users could participate in the process of performing tasks (acquire a sense of engagement). However, robots can only execute some simple tasks when grasping the use of laser interaction in existing research. There is still little discussion about how to further convey complex user intentions and control the robot to complete complex tasks using a laser pointer.Object affordance held that people perceived the content in the behavioral possibility provided by things . J. GrezThe above research shows that there is a relationship between the object to be operated and the action to be performed. Veronica et al. believed that the research on the intention recognition method based on the idea of object affordance is a new direction to plan more complex tasks using people’s attention and object recognition . HoweverKim et al. presented that the robot could perceive the operator’s attention by understanding the user’s gestures, and the robot would identify the objects concerned by the user and infer the task intentions . Kester The method proposed in this paper will be developed on the WMRA with an embedded board, NVIDIA Jetson TX2. The WMRA mainly consists of a robotic arm, an electric wheelchair, and other components, and the whole robot is powered by two DC 12 V batteries in series, as shown in Object recognition: Affordance information of target objects will be accessed, such as visual appearance and spatial relationships.Point cloud data processing: 3D point clouds in the scene are filtered to acquire the objects’ coordinates utilizing the point cloud library.Semantics identification: Identifying the user instructions by the flicker frequency of the laser spot.Intention reasoning: Inferring the user’s intention based on affordance information and semantics information.Grasping posture: An appropriate grasping posture about task intention for the robot arm is generated.Sub-actions reordering: A task expressed by intention consists of some objects and sub-actions. This module will choose and reorder the sub-actions within an action template library and generalize their trajectory.In our system, a user could activate the interaction by pointing to a target object using a laser pointer, as shown in The touching screen interface enables users to acquire interactive information, such as scene information, user’s intention, confirm and emergency buttons, and operaton of the system. The WMRA will perform the task after the user confirms the robot’s display is the same as their idea.Precise detection of the laser spot is crucial for the WMRA to focus on the target objects and subsequently identify their semantics. Instead of using conventional background difference or template matching methods , we propIn the process of laser interaction, laser spot flashes regularly on the target objects, as shown in Object category and laser semantic mapping label will be output from the YOLOv4 algorithm and processed to generate a 26-dimensional array, which is defined as the input of the SVM. Four semantic images and labels are respectively presented in the first and fourth rows of SVM is known to learn small samples due to the principle of maximum margin ,47. To tAfter solving the weight parameters of SVM, the complete semantic identification module would recognize the interactive information from a laser pointer.The semantic identification module transmits the category information of selected objects to the intention reasoning module, as shown in The international classification of functioning, disability, and health (ICF) guideline describes the necessary tasks to maintain the lives of people in a home environment. Therefore, we construct a knowledge base dataset about household tasks by extracting the executed object and action labels from the ICF, as shown in In order to formalize affordance between objects, we propose an intention reasoning method based on the conditional random field (CRF). The CRF is a machine learning algorithm that can predict implicit random variables The CRF algorithm consists of nodes and edges. As depicted in The object and action sets are represented as To solve the weight parameter, we use the log-likelihood equation and the improved iterative scaling method. The log-likelihood equation is expressed as:The prediction of OAIN is the Viterbi algorithm, which can quickly traverse all probability distributions based on the known input sequence and select the maximum probability. The predicted action with the maximum probability can be expressed as:Although we have trained the OAIN using the CRF algorithm, the OAIN may not correctly output the user’s intention when the user’s preference changes. As shown in Q-learning algorithm is a value iteration algorithm based on the Markov decision process in the field of reinforcement learning. The core of Q-learning is making decisions by evaluating the change in Q value. Q value refers to the reward or penalty value after the agent selects the action, and the action triggers a state change for the agent. The Q value will be iterated based on the Bellman equation and stored in the Q table, which consists of state and action.The low-probability intention reasoning results in the traversal process of the Viterbi algorithm being put into the data cache pool so that the user expresses a decision about the intention through laser interaction, as depicted in In our scene, the intention of the user may change over time, which means optimal strategy will not exist. Therefore, we adjust the Bellman equation and embed it into our framework. The iterated equation is expressed as:To perform actions intention sequence continuously, the finite state machine (FSM) is used to logically assemble actions. The FSM is a state transition mechanism, which has the characteristics of state storage and logic coordination. Given the demonstrated trajectories in the template library cannot adapt to the change of object position in the scene, the dynamic movement primitives (DMP) algorithm ,49 is usIn this section, the feasibility of the proposed implicit interaction technology is verified on a WMRA platform. We test the ability of the laser interaction, the intention reasoning model, the learning mechanism, and task execution. Compared with some previous works ,44, our We have deployed YOLOv4 and SVM-based laser interaction on the embedded board, Jetson TX2. Compared to other visual algorithms, such as Transformer , the YOLAs shown in Then, we test the accuracy of SVM for semantic recognition and construct a neural network (NN) for comparison, which includes three fully connected layers, and the activation function is Rectified Linear Unit (ReLU). Before the test, we created a guide that describes some target semantics to be implemented (Each semantic is repeated 20 times in a random order in the guide). Researchers click on the laser pointer to output the semantics and judge whether the result is consistent with the content of the guide. The recognition accuracy is shown in A timing-based method is used as a baseline, which recognizes the laser semantics by calculating the time the laser spot appears. The results show that the SVM and NN methods are much better than the baseline. However, compared to the black-box feature of NN, SVM has a rigorous mathematical theory and is a more transparent classification algorithm. Therefore, we combine the YOLOv4 and SVM algorithms to recognize the laser semantics.In order to compare the ability of intention reasoning with the previous work , which cUnder the same conditions of quantity and type of objects and actions, our work can output 36 single-object intentions and 19 multi-objects intentions. In the other work, the reasoning model could only output the intention with an object and an action, which is low relative to our model. Compared to their work, we can output intentions for several objects and actions. In our opinion, more intentions reflect that the robots have a better comprehension of users and afford users an increased range of options.The main metric for evaluating performance is the number of human-computer interactions in each session. The object in the scene selected by the user for the first time is defined as entering the first session, and the laser semantic “yes” or “no” of the user output decision is defined as human–computer interaction. In the process, the user may judge the target intention and find the desired intention in the first session through multiple human-computer interactions. When the user selects the same objects for the second time, it is defined as the second session, and the user may only need less human-computer interaction time to determine the target intention in the second session. The ideal situation will appear when the number of interactions is one, and it is indicating that the first query to users is their expected intention. As shown in Besides, we also evaluate our method through the intention composed of multiple objects, and to our knowledge, the previous work has not achieved this. As shown in As shown in Furthermore, we record the time spent on intention reasoning. As we can see from To evaluate the execution ability of laser implicit intention interaction technology on the WMRA platform, we record and analyze the process of performing tasks, and compare our method with joystick operation.As illustrated in To improve the control sense, usability, and adaptability of the WMRA, we propose an object affordance-based implicit interaction technology using a laser pointer. A laser pointer could lock the target objects, similar to the human gaze. Therefore, a YOLOv4 and SVM-based laser semantic identification algorithm is proposed to convey user intention intuitively. Moreover, the robustness of the algorithm contributes to the automatic correction of the wrong semantics when the user points to a wrong position because of hand tremors. Next, we were inspired by the cognitive psychology of human interaction processes and designed an object affordance-based intention reasoning algorithm, which is based on CRF and Q-learning. The WMRA realized the implicit action intention identification of target objects and learned user preference through the decision of user interaction. Lastly, in order to execute the action sequence in the scene, trajectories of actions were generalized by the DMP and transmitted to the FSM to logically reorder.We selected many objects and conducted the click test of laser interaction. The results show that the lowest success rate of correctly selecting a desired object is 92%, and the highest is 99%. We inferred that the impacts on the success rate are the surface curvature and material, which is consistent with the discussion in . HoweverBased on the experiments and results, we believe that our research could reduce the burden of interaction between disabled users and assistive robots. Users only need to sequentially click on the target objects, and then they can wait for the robot to execute automatically. Due to security considerations, the research did not include human-related tests. However, in our opinion, the less participation during the task execution, the fewer the users’ interaction burden. Overall, this paper realizes the conveyance of complex task intention through simple interactive operation and executes it, which lays a technical foundation for the rapid practical application of the WMRA.This paper explores an implicit interaction system that can assist users in completing household self-care tasks, which can reduce the interaction burden in the operation of the robot. However, some limitations that affect our research should be noted. For safety reasons, we have not conducted tests on older adults or other individuals, which limits the diversity of experimental samples. To achieve interactive experiments between robots and older adults, the reliability of the proposed system and the psychological pressure that robots bring to users, such as the trusted testing between users and robots , need fuBased on the above limitations, we should not only focus on the execution ability of the robots but also consider the cognitive and decision-making burden of users during the robot operation process. These provide ideas for future research. As reviewed by , Ergonom


The entity–relationship joint extraction model plays a significant role in entity relationship extraction. The existing entity–relationship joint extraction model cannot effectively identify entity–relationship triples in overlapping relationships. This paper proposes a new joint entity–relationship extraction model based on the span and a cascaded dual decoding. The model includes a Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a relational decoding layer, and an entity decoding layer. The model first converts the text input into the BERT pretrained language model into word vectors. Then, it divides the word vectors based on the span to form a span sequence and decodes the relationship between the span sequence to obtain the relationship type in the span sequence. Finally, the entity decoding layer fuses the span sequences and the relationship type obtained by relation decoding and uses a bi-directional long short-term memory (Bi-LSTM) neural network to obtain the head entity and tail entity in the span sequence. Using the combination of span division and cascaded double decoding, the overlapping relations existing in the text can be effectively identified. Experiments show that compared with other baseline models, the F1 value of the model is effectively improved on the NYT dataset and WebNLG dataset. Entity–relationship extraction means to extract entity relationships from unstructured text ,2 and coThe two primary kinds of existing entity–relationship extraction methods are the pipeline extraction method and the joint model extraction method . Among tIn this paper, we propose a novel joint entity–relationship extraction model based on a span and cascaded dual-decoding method (SCDM) to solve the problem of overlapping relationship. First, we use the Bidirectional Encoder Representations from Transformers (BERT) pretrained language model to process the text, convert them into the tokens, and then use the random span division mechanism to divide the tokens to form span sequences. Second, we perform relation decoding on span sequences in combination with a predefined set of relation types to obtain the relation types that exist between span sequence entities. Third, integrating the relationship type obtained by the relationship decoding layer for entity decoding, in the entity decoding part, we first obtain the head entity and then obtain the tail entity. The head entity is decoded by fusing the relation type feature, and the tail entity is obtained by fusing the relation type feature and the head entity. Through the above steps, the model proposed can effectively solve the overlapping relationship, and use the dual-decoding mechanism, which can improve the training speed and reduce the error transmission.A novel joint entity–relationship extraction model, SCDM, based on the span and a cascaded dual decoding are proposed. Different from existing methods, this paper performs the span division after the BERT preprocessing to form span sequences, and subsequent decoding tasks are performed within the span sequences. The issue of overlapping relations in the text is successfully resolved.This method uses a cascaded decoding mechanism, which has the advantages of a fast learning speed and a reduced error transmission. First, it decodes the relationship between entities in the span sequences, and then it decodes the entities in the span sequences to obtain the head entity and the tail entity. The relationship type information obtained by the relation decoding layer is fused when decoding the head entity, and the relationship type information and the acquired head entity information are fused when decoding the tail entity.Through the analysis of the NYT dataset and the WebNLG dataset, when compared to the current models, the methodology suggested in this research produced the best results.Overall, the significant contributions of this paper are as follows:An essential task in the field of natural language processing (NLP) is extracting entity connections from unstructured text, which is also a necessary step in building a knowledge graph to provide support for downstream related tasks ,13. At pThe relationship extraction task is divided into two separate subtasks using the pipeline extraction approach. Nayak T. et al.  introducThe pipeline method does not need to manually construct features and has high accuracy, so it is widely used, but because the error in the entity recognition task may be transmitted to the relationship classification task, it causes the problem of error propagation . TherefoIn recent years, Shang Y. et al.  introducThis paper is also aimed at the problem of overlapping relationships in the text. Different from Ma L. et al. , the metThis section describes the joint entity–relationship extraction model proposed in this paper based on the span and a cascaded decoding. Compared with the original joint model of entity–relationship extraction, if there are overlapping relationships in the text, the effect of the entity–relationship extraction will not be ideal. On the whole, the model is divided into three layers, a BERT decoding layer, a relationship decoding layer, and an entity decoding layer. The entity decoding layer includes two parts: the head entity extraction and the tail entity extraction. In this model, we first input the text into the BERT encoding layer, use the BERT-BASE-CASED pretrained language model to convert the text into vectors, and randomly divide the vectors to form span sequences. Then, we use linear layers to get the type of relationship between entities within the span sequences. Finally, the span sequences information is combined with the relationship type feature to extract the head entity and tail entity in the span sequence. The model is shown in Where blue represents the span sequences, and the number of blue-colored blocks represents the size of the span sequence; yellow represents the relation type obtained by the relationship decoding layer; orange represents the predefined relation type, which contains 24 relation types in the NYT dataset, and the WebNLG dataset contains 246 relationship types; green indicates the head entity obtained by the entity decoding layer. In this model, we use the BERT encoder to generate tokens and divide them into span sequences; we use the relation decoder to get the relations in the span sequence; and we use the entity decoder to get the head and tail entities.We use the BERT pretrained language model to encode texts to capture semantic information between texts. The main task of the encoding layer is to encode the text information into matrix vectors. The encoding layer uses the BERT pretrained language model to obtain the semantic features of the sentence. The semantic features of the sentence are expressed as i-th token in the processed vectors is represented as formula (1):Firstly, the input text sequence is represented as word vectors through the embedding layer, and the n) represents the size of the span sequence formed after the division. The relationship decoding layer decodes the relationship type between entity pairs within span sequences. Given a predefined set of relationship types, as shown in formula (5):n is the size of the relation type Since there are many kinds of relationships between entities in the span sequences, decoding the relationship type in the text is the main task of this layer. Determining the relationships that exist in span sequences is similar to a multilabel classification problem . Span diWe set the relationship filtering threshold. If the value of The main task of the head entity extraction is to extract head entities in relation types, which are obtained from the relation decoding layer. First, the span sequences and relation type information are fused; then, they are input into the Bi-LSTM neural network to obtain the information in the span sequences, where the output representation of the hidden layer of the Bi-LSTM neural network is shown in formula (8):The probability of the head entity in the relation type to which the entity belongs in the span sequences is determined using the softmax activation function. As shown in formulas (9) and (10):The main task of the tail entity extraction is to extract tail entities in relation types. First, the span sequences, head entity information, and relationship type information are fused; then they are input into the Bi-LSTM neural network to obtain the information in the span sequences, where the output representation of the hidden layer of the Bi-LSTM neural network is shown in formula (11):Relationship categorization and entity recognition are both multiclassification issues, so the cross-entropy loss function is used in the SCDM model. The relationship decoding layer, the head entity extraction, and the tail entity extraction’s loss functions are added to create the overall model’s loss function.We evaluated the method proposed in this paper on two datasets, the NYT dataset  and the The model proposed in this paper used the PyTorch framework and NVIDIA GeForce RTX 3090 (24 G) graphic processing unit (GPU), and its hardware and software are shown in The experiment in this paper used the BERT-BASE-CASED pretrained language model. The word embeddings had a dimension of 300 and were randomly initialized. The location embedding dimension was 20. The training parameter settings are shown in This paper used two methods to evaluate model performance: partial match and extract match. The partial match method  states tThe evaluation indicators used in this experiment were precision, recall, and F1-score. The latter is the most important evaluation parameter in entity–relationship extraction.CopyRe  proposedGraphRel  proposedCopyRL  proposedCopyMTL  proposedWDec  proposedAttentionRE  developeCasRel  proposedDualDec  proposedA comparison between the model in this research and the model from more recent years was conducted. Here are the comparative models:The comparison between the model used in this paper and the baseline model on the NYT (N) dataset and the WebNLG (W) dataset yielded the findings shown in To investigate how the model in this research affected various relationship types, we also conducted comparative experiments. Since there were three relationship types in the text, normal, entity pair overlap and single entity overlap, comparative experiments were conducted on different relationship types to test the precision, recall, and F1 score, respectively, as shown in Our model was compared with existing models in the N dataset and the W dataset, which were CopyMTL , WDec 3 AttentioAs shown in According to the analysis of existing entity–relationship extraction methods, this paper proposed a joint entity–relationship extraction model based on the span and a cascaded dual decoding. The model first divided the word vectors based on the span after the BERT pretraining model to form new span sequences and then decoded the relationship type between entity pairs in the span sequence. Finally, the span sequences information was combined with the relationship type feature, and the head entity and tail entity in the span sequences were extracted using the Bi-LSTM neural network. Experimental results showed that the model achieved certain results. The next step will be to study the span sequences formed after the division and to study the relationship between entities in the span sequence and the possible relationship between span sequences, so as to achieve better extraction results when the text contains complex overlapping relationships.


The spatially resolved interrogation of a Fabry–Perot ultrasound sensor using a laser beam focused through a multimode fiber is demonstrated. To scan the beam across the sensor as required to read it out, optical wavefront shaping was employed to compensate for the scrambling of light in the fiber. By providing a means to map ultrasound through inexpensive, lightweight fibers, this could lead to new ultrasonic and photoacoustic imaging systems, such as endoscopes and flexible handheld probes. Fabry–Perot (FP) ultrasound sensors (FPUSs)Applying this measurement process at multiple locations by scanning a readout beam across an FPUS allows broadband ultrasonic mapping with microscale spatial sampling and high ,A schematic of the setup is shown in To focus the light onto a particular position on the FPUS, a GA was used maximize the brightness at this position. To do so, first, a set of random phase patterns were displayed on the SLM. A subset of the phase patterns maximizing the brightness at the target location were selected and mixed to generate new phase patterns to display. This procedure was iterated, resulting in the gradual formation of a focus at the target region. To reduce the optimization time, the SLM elements were grouped into a smaller number of pixel groups, within which the same phase modulation was applied. To maximize the achievable “power ratio” , the illuminated region on the SLM was chosen such that the number of illuminated pixel groups was equal to the number of modes supported by the MMF.The procedure was used to focus light onto the FPUS at a location co-incident with the center of the camera's field of view. To examine the change to the field, images were recorded before and after WFS. The image obtained prior to WFS is plotted in To explore focusing at different locations and obtain an indication of the achievable power ratio, the above experiment was repeated in different conditions, including with different numbers of GA iterations, and different target locations for the focus. It was found that near-diffraction-limited foci could be produced anywhere in the field of view. The power ratio typically increased with the number of GA iterations, with a maximum value of ∼60% (after ≳50 iterations). The remainder of the optical power formed an effectively random low-intensity background speckle pattern.After focusing the beam onto the FPUS, measuring ultrasound required determining an appropriate FPUS bias wavelength, λb Ref. , at the To address this need, a four-step procedure was followed to obtain ITFs and bias wavelengths valid for spots produced via WFS. First, prior to WFS, an ITF, The above procedure was applied to measure an ITF and a bias wavelength at a chosen point on the FPUS. To measure the transmitted power, the camera was temporarily repositioned behind the FPUS. To measure To demonstrate spatially localized ultrasound measurements, an experiment was devised that involved sampling a known spatially varying ultrasonic wavefront at four locations. A schematic of the setup is plotted in v was the speed of sound in water. As a reference, a waveform was also captured using random speckle illumination.The setup was used to measure ultrasonic waveforms after focusing the beam onto the FPUS at an appropriate bias wavelength using the methods described above. The measurement points were chosen to be positioned along a line perpendicular to the axis of rotation of the transducer. In this arrangement, the difference in the times of arrival of the pulses expected at each point was One of the raw ultrasonic waveforms obtained using a WFS-formed optical spot is plotted in The resulting background-subtracted waveforms are plotted in The presented experiments show that an FPUS can be interrogated through MMF, in turn allowing spatially localized US measurements. As well as this main finding, certain features of the experiments highlight directions for further research.One feature of the experiments is that they were slow, but they could be sped up to enable high-resolution imaging. Specifically, focusing light onto the FPUS took tens of minutes, limiting the number of measurements, and reducing the practicality of imaging. The limiting factors were the pattern rate of the SLM (<10 Hz pattern rate) and the iterative nature of the GA. Both could be addressed. For instance, swapping the SLM for a faster digital-micromirror device (DMD)A second feature of the experiments was that, while the detected US waves were partially localized to the WFS-formed interrogation spots, a background signal was also observed due to the uncontrolled portion of the speckle. This effectively resulted in “crosstalk” between the ∼115 measurement channels in the MMF, only four of which were used to make the actual measurements. Here, a simple approximate subtraction was used to partially suppress this crosstalk. However, more sophisticated approaches could be used to reduce it more completely. For example, if most of the available channels in the MMF were characterized by measuring the TM, then techniques analogous to those used in multiple-input multiple-output communications systemsA final feature of the experiments was that the MMF was held stationary throughout. The purpose was to avoid any complications due to the TM of the MMF changing in response to environmentally induced fiber strain.In summary, we showed that it is possible to spatially map US fields using an FPUS interrogated through MMF. In this proof-of-concept work, the number of detection points was low due to practical constraints. However, speeding up the system could allow tractable ultrasonic imaging with >1000 channels. Uncontrolled background light produced inter-channel crosstalk. However, this too could be mitigated, increasing the signal fidelity.By providing a means to map ultrasonic fields through narrow, inexpensive, and lightweight fibers, this could lead to the development of a range of new ultrasonic and photoacoustic imaging systems. As for other devices based on MMF, such systems could benefit from compatibility with multi-modality imaging through a single fiber. This interrogation scheme may provide additional advantages to FPUS-based imaging, as control over the structure of the interrogation beam could lead to higher sensitivity


In many healthcare applications, datasets for classification may be highly imbalanced due to the rare occurrence of target events such as disease onset. The SMOTE (Synthetic Minority Over-sampling Technique) algorithm has been developed as an effective resampling method for imbalanced data classification by oversampling samples from the minority class. However, samples generated by SMOTE may be ambiguous, low-quality and non-separable with the majority class. To enhance the quality of generated samples, we proposed a novel self-inspected adaptive SMOTE (SASMOTE) model that leverages an adaptive nearest neighborhood selection algorithm to identify the “visible” nearest neighbors, which are used to generate samples likely to fall into the minority class. To further enhance the quality of the generated samples, an uncertainty elimination via self-inspection approach is introduced in the proposed SASMOTE model. Its objective is to filter out the generated samples that are highly uncertain and inseparable with the majority class. The effectiveness of the proposed algorithm is compared with existing SMOTE-based algorithms and demonstrated through two real-world case studies in healthcare, including risk gene discovery and fatal congenital heart disease prediction. By generating the higher quality synthetic samples, the proposed algorithm is able to help achieve better prediction performance (in terms of F1 score) on average compared to the other methods, which is promising to enhance the usability of machine learning models on highly imbalanced healthcare data. Imbalanced data classification, which refers to the problem of classification when there is an uneven distribution of classes in the training dataset, has been encountered in many healthcare applications, and it is considered as an important topic to discuss in the biomedical data mining area –4. In clThe performance of classification models built on highly imbalanced datasets, can be improved either at the algorithmic or data pre-processing level . At the Two commonly used resampling methods are downsampling and oversampling . The dowk-nearest neighbors that are close in vicinity, drawing a line between the selected samples and generating a new sample at a random point along the line. Although the SMOTE algorithm has the ability to generate more minority data for enhancing the classification accuracy, there are some limitations with the quality of the generated samples. First, SMOTE generates data based on the nearest neighbors acquired from the k-nearest neighbor algorithm, but it is usually hard to identify the real valuable neighbors when the classes of data are not well separated . For the other K values, the proposed SASMOTE algorithms always have higher average F1 scores than the other SMOTE-based algorithms, which indicates the robustness of the proposed algorithm with respect to the value of K. When K is relatively large , the SASMOTE algorithms with inspection have much lower standard deviations than the other resampling algorithms, which indicates the importance and effectiveness of quantifying the uncertainty of resampled data via inspection when K is large.Table N values. It showcases that SASMOTE methods on average has better performance than other SMOTE-based algorithms under different settings of N. Even though they do not achieve the best F1 score in the case of Table T). SASMOTE performs better than SASMOTE without visible neighbors on average when the threshold is between 50% to 60% while it performs worse when the threshold is between 75% to 90%. This experiment suggests that SASMOTE without visible neighbors needs higher threshold of uncertainty since the low-quality samples can be generated from the invisible nearest neighbors. SASMOTE, on the other hand, needs lower threshold of uncertainty because the samples generated from the visible neighbors have higher quality. The average performance of both methods increases with respect to the threshold of uncertainty score first and then decreases. This is due to the high threshold of uncertainty score can lead to smaller number of training samples, which potentially influences the classification accuracy. Therefore, 50% is the optimal threshold of uncertainty score for both SASMOTE and SASMOTE without visible neighbors algorithms.Table The resampling techniques, especially SMOTEs, are popular for solving imbalanced data classification problem in healthcare. However, existing SMOTE algorithms may generate “uncertain” samples that fall into to the wrong class due to the lack of selection of nearest neighbors used for sample generation and the lack of inspection of generated samples. To close this gap, this paper proposes a self-inspected adaptive SMOTE (SASMOTE) algorithm. The proposed algorithm introduces an adaptive nearest neighborhood selection algorithm to identify the “visible” neighbors for generating more accurate minority class samples. An uncertainty via inspection approach is further developed in the proposed method to measure the quality of resampled data and filter out the low-quality ones. The proposed method is capable of generating high-quality resampled data, further improving the performance of machine learning models on highly imbalanced healthcare data and advancing the development of biomedical data mining.By applying the proposed SASMOTE method to two real-world healthcare datasets, risk gene discovery and fatal congenital heart disease prediction, the SASMOTE method on average has better performance than the other SMOTE-based resampling algorithms, which demonstrates the advantages of using the “visible” neighbors and uncertainty via inspection. By comparing the SASMOTE method with and without self-inspection algorithm, the uncertainty via self-inspection algorithm is found to be critical for reducing the variance in resampling and improving the robustness in classification. Furthermore, the SASMOTE method shows advantage on the average performance over other SMOTE-based resampling algorithms under different imbalanced ratios, and the advantage is more significant when the dataset is highly imbalanced. Through the sensitivity analysis, we explored the effects of hyperparameters, including the number of nearest neighbors, number of generated samples, and threshold of uncertainty score, on the SASMOTE performance. The SASMOTE method is preferred when the number of nearest neighbors is relatively large and the threshold of uncertainty score needs to be carefully tuned for achieving better average performance.The two case studies conducted in this study may not be able to cover all types of imbalanced data classification problems in healthcare. In the future, we will apply the proposed SASMOTE method to other applications, such as the rare disease prediction , 29. The


Portable depth sensing using time-of-flight LiDAR principles is available on iPhone 13 Pro and similar Apple mobile devices. This study sought to characterize the LiDAR sensing system for measuring full-field vibrations to support modal analysis. A vibrating target was employed to identify the limits and quality of the sensor in terms of noise, frequency, and range, and the results were compared to a laser displacement transducer. In addition, properties such as phone-to-target distance and lighting conditions were investigated. It was determined that the optimal phone-to-target distance range is between 0.30 m and 2.00 m. Despite an indicated sampling frequency equal to the 60 Hz framerate of the RGB camera, the LiDAR depth map sampling rate is actually 15 Hz, limiting the utility of this sensor for vibration measurement and presenting challenges if the depth map time series is not downsampled to 15 Hz before further processing. Depth maps were processed with Stochastic Subspace Identification in a Monte Carlo manner for stochastic modal parameter identification of a flexible steel cantilever. Despite significant noise and distortion, the natural frequencies were identified with an average difference of 1.9% in comparison to the laser displacement transducer data, and high-resolution mode shapes including uncertainty ranges were obtained and compared to an analytical solution counterpart. Our findings indicate that mobile LiDAR measurements can be a powerful tool in modal identification if used in combination with prior knowledge of the structural system. The technology has significant potential for applications in structural health monitoring and diagnostics, particularly where non-contact vibration sensing is useful, such as in flexible scaled laboratory models or field scenarios where access to place physical sensors is challenging. Ubiquitous computing and powerful sensors available in consumer mobile devices have the potential to displace more expensive measurement, data acquisition, and postprocessing workflows for vibration-based structural health monitoring. Light detection and ranging (LiDAR), implemented in the iPhone 13 Pro ,3,4,5 anMobile devices have evolved to carry a significant number of on-board sensors with the ability to conduct data acquisition and postprocessing on the device itself ,12,13. SThe use of ToF sensors to determine modal characteristics of a structure was demonstrated by Silva et al., who employed a Microsoft Kinect range camera to capture dynamic point clouds at a 30 Hz sampling rate to accurately identify the first three modes and frequencies for a suspended rectangular steel plate . BenedetStudies of iPhone LiDAR for performing 3D reconstruction of static scenes have concluded that the sensor and various scanning apps can produce 3D models with an absolute accuracy of 1 cm for objects on the order of a few meters . While dThe rest of this paper is organised as follows. The rear-facing LiDAR sensor on the iPhone 13 Pro is in the lower right corner of the camera cluster, as shown in A good graphical representation of the matrix of points emitted from the iPhone LiDAR, as captured by a near-infrared camera, is provided by Teppati et al. in . Allain (i)We characterise the LiDAR sensing system available on the iPhone 13 Pro and similar Apple devices regarding its static measurement properties, exploring its accuracy with different phone-to-target distances, noise floors, and lighting conditions.(ii)We define the dynamic characteristics and capabilities of the sensor regarding dynamic accuracy, range, and sampling rate effects, and further relate these to applications and limitations of LiDAR in modal analysis.The majority of the studies on LiDAR involve dedicated sensors specifically built for particular applications; these are generally quasi-static, i.e., there is no significant movement of the measured target during the LiDAR operation. The problems we address in this study are twofold:To this end, in  sensors ,36,37.d from the plate. First, we examine the static characteristics of the LiDAR sensing system of the iPhone 13 Pro. The experimental arrangement comprises a dark rectangular plate positioned with backside support, and the phone is affixed to a movable support at a distance Each LiDAR reading comprised a depth map consisting of a grid measuring RKit API . In all d and compared the statistical properties of the measurements at the central point of the captured frames, i.e., Considering that the LiDAR operates based on the time-of-flight (ToF) principle, it is to be expected that the measurement accuracy would vary with the distance between the sensor and its target. To investigate this effect, we conducted a static experiment with multiple phone-to-target distances One notable characteristic of LiDAR sensors is their ability to perform well in dark or poorly illuminated environments, which is mostly related to the detection probability phenomenon in the SPAD ,40. In oIn order to assess the dynamic characteristics of the LiDAR sensing system, we employed a linear air-bearing shaker (APS 113-AB) to mount the rigid plate mentioned in The depth information collected with the iPhone’s LiDAR sensing system is internally combined with RGB data, and is obtained at a sampling frequency of 60 Hz . ConsequAn initial investigation focused on assessing the accuracy and noise characteristics of the LiDAR sensing system when measuring dynamic targets. To this end, we subjected the target plate to a forced harmonic oscillation at various frequencies while simultaneously recording the motion using both the LiDAR sensor and the laser transducer. Both sensors were positioned at a distance of 35 cm from the target plate. After analyzing the normalized statistical properties of the LiDAR measurements, as illustrated in Furthermore, a comparison between the LiDAR sensing system and the laser displacement transducer (LDT) was conducted for a broadband signal consisting of frequencies ranging from 1 Hz to 60 Hz. In this scenario, the shaker was provided with a target spectrum as the input and measurements were simultaneously obtained from both the LiDAR sensing system (at the centre of the field of view) and the LDT. The recorded time histories are presented in To further investigate the measurement accuracy of the LiDAR sensing system, we examined the impact of varying phone-to-target distances for a fixed oscillation frequency of 4 Hz, which falls within the reliable RMS identification range of the LiDAR. The results presented in For distances shorter than We proceeded to evaluate the LiDAR sensing system’s sampling rate properties. The sampling interval was examined using a Monte Carlo (MC) approach that encompassed eighteen individual measurements each lasting approximately 83 s, resulting in a total measurement duration of approximately 1500 s. The mean sampling rate obtained from these measurements is Subsequently, we investigated the frequency content of the forced oscillation scenario with an oscillating frequency of  theorem , when coUpon further investigation, we hypothesized that the accurate internal LiDAR sampling rate would amount to h denotes the convolutional kernel that determines the type of interpolation. Three distinct strategies were tested: sample-and-hold (S-A-H), linear interpolation, and smoothed quadratic interpolation. The convolutional kernels h as functions of the time step To validate the assumption of upsampling, synthetic single harmonic signals denoted as For further analysis, we selected the upsampling results corresponding to forced oscillation frequencies of h can aid in explaining the magnitude of the peaks observed in the signal’s power spectral density (PSD). Their frequency domain representation can be interpreted as a linear transfer function between the original signal and the upsampled signal, providing insights into the peak amplitudes. Subsequently, we compared the frequency response of each convolutional kernel to the spectral amplitudes of the expected frequency In addition to assessing the presence of aliases in the upsampled signal, the convolutional kernels In practical measurements, there is no straightforward method to revert the upsampling process and accurately distinguish between the true frequencies and the aliases. The current LiDAR measurements are most effective for signals with frequency content limited to We now present an application of the iPhone’s LiDAR sensing system for the modal identification of a steel cantilever structure with a 16 mm by 2 mm cross-section and a height of The modal properties of the cantilever were initially determined using the LDT only. By positioning the laser transducer at a height of x) direction that encompasses readings across the height of the cantilever. Subsequently, the top-most point of the cantilever was determined as the first z coordinate containing readings compatible with the static distance between the cantilever and the iPhone, which is approximately The collected depth map covers the entire LiDAR field of view, as depicted in The processed depth map statistics depicting the envelope and root mean square (RMS) of readings across the cantilever height are displayed in Next, we proceeded to compare the power spectral density of the laser displacement transducer measurements at a height of After completing the preprocessing steps and eliminating the bottom and tip cantilever parts, the depth dataset contained 119 of the initial 171 measurement points in a regular grid spanning from Modal analysis was conducted using the Stochastic Subspace Identification (SSI) method ,45. Instf, damping ratios The stabilization diagram for one of the Monte Carlo cases is illustrated in The results of the modal analysis in z is the height coordinate, H is the cantilever total height, and i obtained as the i-th solution of the equation In each of the 1000 Monte Carlo iterations, the mode shapes for all four modes were estimated at the five locations defined by each specific sample from the Latin Hypercube. ess (GP)  for fittStrong agreement is observed between the analytical mode shapes and those identified by the SSI, particularly for the first, second, and fourth modes. By employing GP fitting, we quantitatively assessed the uncertainty associated with each mode based on the optimal identification of noise while considering the scattered modal points identified through SSI. Among the four modes, mode 1 exhibits the smallest noise standard deviation, amounting to In this paper, we have investigated the characteristics of Apple’s iPhone 13 Pro LiDAR sensing system while focusing on its optimal measurement distance, noise properties, and sampling rate properties. Additionally, an application of the sensor to the modal identification of a steel cantilever structure is presented.Previous studies have highlighted the utility of LiDAR technology for static measurements. Our results indicate that a minimal phone-to-target distance of 30 cm yields optimal measurement values and an acceptable signal-to-noise ratio in static conditions. However, a counterintuitive finding was that accuracy decreases and uncertainty increases in poorly illuminated conditions. This is likely due to Apple’s internal combination of RGB data and depth information to generate a dense depth map, which adversely affects LiDAR readings under dark conditions. In practical applications, various additional external factors can potentially impact the quality of the acquired depth maps, including environmental conditions such as rain and fog. To enable the deployment of such a system, further investigation of these topics is necessary and constitutes a crucial area for future research.For dynamic measurements, good displacement RMS values were obtained for phone-to-target distances up to 200 cm, indicating an optimal LiDAR measurement range between 30 and 200 cm. Nevertheless, even within this range the noise content in LiDAR measurements is significantly higher than in dedicated displacement sensors such as laser displacement transducers. An important outcome is that while LiDAR data are commonly output at a sampling frequency of 60 Hz to match the RGB camera, these data in fact have a sampling rate of 15 Hz. Upsampling is performed to match the target sampling frequency, introducing aliases to the frequency content of the depth measurements. Additionally, we observed that while the upsampling procedure is frequency-dependent, it can be approximated effectively via simple linear interpolation.To demonstrate the LiDAR sensing system’s properties, an experiment involving the modal identification of a steel cantilever was conducted using covariance-driven Stochastic Subspace Identification in a Monte Carlo setting while providing confidence intervals for all modal properties. The identified natural frequencies matched those obtained from the laser displacement transducer readings with an average error of 1.9%, while the damping ratios significantly differed from those obtained from logarithmic decrement analysis, likely due to the high level of noise in the depth measurements. The mode shapes were further processed using a Gaussian process fitting, providing continuous averages and uncertainty ranges. After comparing them with an analytical counterpart model of the cantilever, high Modal Assurance Criterion (MAC) values were observed for the first two mode shapes, which lie below the Nyquist frequency of the LiDAR measurements. The two higher modes were identified as aliases in the true frequency content, exhibiting lower MAC values and higher standard deviations, again likely due to noise.The investigated sensor holds potential applications in the fields of Structural Diagnostics and Structural Health Monitoring of civil and mechanical systems. In its current state, it enables efficient and portable noncontact displacement measurements, overcoming many logistical and operational issues associated with more traditional sensing strategies. In the context of civil engineering applications, the present limitations of phone-to-target distance and field-of-view dimensions confine the practical utility of this sensor technology to flexible structural components, where the analysis of frequency content is of primary importance. Such components include structural cables in suspension and cable-stayed bridges as well as external post-tensioning tendons. The LiDAR sensing system can find a diverse range of applications in laboratory settings, particularly in the study of slender scaled structural models. The noncontact property is of particular advantage in such contexts, as it avoids any potential alterations to the structural dynamic properties. Enhancements to the sensor, such as increased phone-to-target distances and improved signal-to-noise properties, along with the provision of raw depth data before RGB interpolation, could further enhance its usefulness for monitoring purposes.


Three-dimensional laser scanning has emerged as a prevalent measurement method in numerous high-precision applications, and the precision of the obtained data is closely related to the intensity information. Comprehending the association between intensity and point cloud accuracy facilitates scanner performance assessment, optimization of data acquisition strategies, and evaluation of point cloud precision, thereby ensuring data reliability for high-precision applications. In this study, we investigated the correlation between point cloud accuracy and two distinct types of intensity information. In addition, we presented methods for assessing point cloud accuracy using these two forms of intensity information, along with their applicable scopes. By examining the percentage intensity, we analyzed the reflectance properties of the scanned object’s surface employing the Lambertian model. Our findings indicate that the Lambertian circle fitting radius is inversely correlated with the scanner’s ranging error at a constant scanning distance. Experimental outcomes substantiate that modifying the surface characteristics of the object enables the attainment of higher-precision point cloud data. By constructing a model associating the raw reflectance intensity with ranging errors, we developed a single-point error ellipsoid model to assess the accuracy of individual points within the point cloud. The experiments revealed that the ranging error model based on the raw intensity is solely applicable to point cloud data unaffected by specular reflectance properties. Moreover, the devised single-point error ellipsoid model accurately evaluates the measurement error of individual points. Both analytical methods can be utilized to evaluate the performance of the scanner as well as the accuracy of the acquired point cloud data, providing reliable data support for various high-precision applications. High-precision terrestrial laser scanning (TLS) has gained prominence in various fields due to its capability to efficiently and rapidly acquire precise 3D data over extensive areas. With the advancement and optimization of technology, acquiring high-precision data has become increasingly crucial in diverse fields such as precision mapping ,2, enginScanner mechanism: factors include the stability of laser emitters, performance of receivers , signal Environmental factors: atmospheric conditions, lighting conditions, and the reflective properties of surrounding objects can influence laser propagation and consequently the accuracy of point cloud data;Surface and scanning geometry: the color, texture, material, and roughness of an object’s surface, as well as the scanning distance and incident angle, can cause variations in the reflected laser intensity, impacting the ranging accuracy.Scanner placement: unstable installation or imprecise positioning of the scanner can result in measurement errors.However, the accuracy of point cloud data is subject to numerous factors, which can be classified into four primary categories:To meet the high-precision requirements, researchers continuously strive to optimize laser scanning technology, enhancing data quality and application effectiveness. Among the four categories of errors previously mentioned, measurement errors arising from scanner mechanism can be eliminated or reduced through calibration methods ,14. WithLichti et al.  investigAs the utilization of TLS grows in fields demanding high precision, such as deformation monitoring and industrial manufacturing, a comprehensive understanding of 3D laser scanner performance is essential. To further enhance the quality of the acquired point cloud data and precisely assess the measurement uncertainty of the point cloud, we used the Zoller + Fröhlich Imager 5016 as an example to extensively examine the effect of intensity information on point cloud accuracy. We developed a single-point error ellipsoid model based on the raw intensity. Additionally, to address the issue of some scanners not providing access to raw intensity data, we employed the Lambertian reflectance model to examine the relationship between percentage intensity and ranging accuracy. The experimental findings can be applied to optimize scanner placement, enhance the accuracy of a collected point cloud, and accurately evaluate the spatial errors of point cloud data.The organization of this paper is outlined as follows. The scanner used in this study was the Z + F Imager 5016, a compact and lightweight device designed for the rapid acquisition of high-resolution point cloud data. As is stated in reference , the parTo investigate the correlation between intensity and point cloud accuracy for various materials, this study employed three industrial wooden panels painted with a layer of concrete on their surfaces as scanning targets. The dimensions of these panels measured 1.2 m × 1.0 m. Two of the test panels featured a uniform application of white and yellow paint on the concrete surface, as depicted in To facilitate the acquisition of point cloud data from the three panels at varying distances and angles, a mobile workstation was developed to house the panels, as shown in The data acquisition scheme, illustrated in The precision of the point cloud was determined by fitting the target point cloud data to a plane, followed by computing the root mean square error (RMSE) of the plane residuals. Plane fitting employs the method of total least squares, minimizing the orthogonal distance from the point to the fitted plane . Before The calculation of plane fitting errors was obtained by computing the vertical distance from the points to the fitted plane, as shown in The Lambertian reflectance model is a classic model used to describe the reflection behavior of light on surface materials . This moPoint cloud data typically contain location information for each point as well as other related attributes, one of which is intensity. Intensity refers to the energy value of the light emitted by the LiDAR system when it strikes an object’s surface and reflects back. This intensity exhibits a direct proportionality to the power of the backscattered signal. Intensity values can provide information regarding the surface properties of an object, such as color, material, and roughness. The intensity recorded by scanners can be categorized into two types. The first category is the raw intensity, which corresponds to the intensity directly measured by the device. The measurement range may vary among different devices, and some manufacturers may not permit users to export these data. The second category is the normalized intensity, which standardizes the raw reflectance intensity values into a specific range, such as 0 to 1 or −1 to 1. For different scanners, the normalization methods for raw reflectance intensity may differ. For scanners that are unable to access raw intensity data, the normalized percentage intensity data and the Lambertian reflectance model can be utilized to analyze point cloud accuracy.The relationship between the radiant flux reflected within the solid angle and the radiant flux reflected in the surface normal direction for a Lambertian surface is as follows :(3)Iθ=I0OA represents the reflectance intensity OB represents the reflectance intensity In The The error ellipsoid is a geometric shape used to describe the uncertainty of a position or state. When a scanner records the coordinates of a point, there will always be errors caused by distance and angle measurements. By utilizing the error ellipsoid, the uncertainty in spatial location of a point caused by these errors can be effectively described. For any point in Cartesian space, its spatial uncertainty can be represented as Equation (6):When a scanner records the coordinates of a point, it usually uses polar coordinates. During subsequent processing, the polar coordinate system is converted into the Cartesian coordinate system:x, y, and z are the Cartesian coordinates of a point, In this equation, K can be obtained by taking the total differential of Equation (7).Without considering the influence of atmospheric and other factors, the uncertainty of a point is mainly affected by distance measurement and angle measurement, and follows a normal distribution. Based on the error propagation law, the coefficient propagation matrix K, the covariance matrix of the point’s coordinates in the Cartesian coordinate system can be calculated:The range and angle measurements of the scanner are uncorrelated; thus, the covariance matrix k is the amplification coefficient of the error ellipsoid, determined by a given probability. The covariance matrix Q. Then, Equation (11) can be converted intoHence, the error ellipsoid for a point in the Cartesian coordinate system can be described asIn this equation, Rf denotes the raw intensity, and In most instances, the scanner’s angle error can be derived from values provided by the manufacturer or calculated following scanner calibration. The ranging error provided by the scanner manufacturer is a rather general description, but there is a specific functional relationship between ranging error and the raw intensity. By constructing a random model for raw intensity and ranging error, the ranging error can be determined through the raw intensity value.During the experimental data collection with the Z + F Imager 5016, both percentage and raw intensities for each point were concurrently documented. It became clear that the scanner’s recorded percentage intensity is more than a simple normalization of the raw intensity. As is illustrated in Using the experimental plan introduced in As is demonstrated in The Lambertian circle fitting results indicate that, in the absence of considering the vertical incidence of the laser beam, the experimental panels for all three materials conform to Lambertian properties. The white painted surface displays the largest Lambertian circle fitting radius, while the concrete surface exhibits the smallest. Additionally, the Lambertian circle fitting radius decreases progressively as the scanning distance expands. By combining This finding highlights the significant impact of the returned beam’s energy magnitude on point cloud accuracy for laser scanners. In order to obtain point cloud data that satisfies the accuracy requirements of specific applications, a comprehensive evaluation of the scanner’s performance is crucial. By analyzing the scanner’s ranging error at different ranges and incident angles, a preliminary understanding of scanner performance can be obtained.By analyzing the ranging errors of the scanner at various distances and incident angles, a preliminary understanding of the scanner’s performance can be achieved, determining whether the scanning accuracy of the scanner can meet the precision requirements of meticulous applications.Moreover, in some precision applications, such as deformation monitoring, if circumstances allow, the reflective properties of the surface of the object being scanned can be altered by applying an appropriate coating. Through this approach, the Lambertian characteristics of the scanned object’s surface are enhanced, and the ability of the object’s surface to reflect light beams through diffuse reflection is strengthened, thereby enabling the acquisition of high-quality, high-precision point cloud data.High-precision point cloud data are crucial for a variety of precision applications in 3D laser scanning technology. To obtain such data, it is essential to understand the performance of the scanner in use and to scientifically set up the scanner based on its capabilities. Moreover, modifying the reflective properties of the scanned object’s surface can enhance the energy of the returned beam received by the scanner. To validate the conclusion that altering surface characteristics can yield higher-precision data, we compared the registration errors of point clouds from the concrete surface and the white painted surface in two different periods using data from a cavern of the main underground plant of a pumped storage power station with a part of the roof painted white . The twoThe first point cloud dataset that was collected is shown in To mitigate the impact of the points representing the anchor rods and the total station prism on registration accuracy, manual removal of these points was implemented in both datasets. Point clouds from the experimental areas in the two periods were individually cropped and registered. Coarse registration utilized pre-positioned target spheres within the cavern, while the ICP algorithm was employed for precise registration. Under identical scanning conditions, higher point cloud data accuracy corresponds to thinner data layers, which then results in lower root-mean-square errors after registration. The experimental findings reveal that painting the cavern vault’s concrete surface white can elevate data intensity by approximately 20%, consequently improving point cloud accuracy. Considering the one-month interval between the two data collection periods, dust and oil fumes accumulated on the vault surface, diminishing the intensity of the second-phase dataset compared to the first. Nonetheless, the white painted surface in the second-phase exhibited a higher intensity than the concrete surface point cloud data in the first phase.Acquiring high-precision point cloud data is crucial for precision applications, and accurate assessment of the data accuracy is of significant importance in such applications, such as model reconstruction and deformation analysis, among others. A rational evaluation of point cloud accuracy aids in comprehending data quality and can function as a quality control measure. By examining point cloud accuracy, potential issues within the dataset, such as whether the data precision meets the application requirements, can be detected, thereby informing decision-making processes.In the data collection experiment outlined in Based on the experimental data, To confirm the single-point error ellipsoid model’s ability to accurately evaluate point cloud data accuracy, point cloud data from an office was gathered using a consistent scanner and sampling settings, as depicted in When employing the error ellipsoid to determine the plane fitting error of an individual point, the assumption is that the point’s true value resides on the fitted plane. The fitted plane’s normalized normal vector is represented by d, from the points on the error ellipsoid to the fitted plane:Calculate the distance, The maximum distance, The scanner’s ranging error is derived from the random model presented in As can be observed in This study analyzed the relationship between two types of reflected intensity and point cloud accuracy, using data obtained from the Z + F Imager 5016 3D laser scanner. A point cloud accuracy analysis method based on the Lambertian reflectance model and per-centage intensity was proposed, which is applicable for assessing the performance of the scanner and the accuracy of the acquired point cloud data in situations where raw intensity information is unavailable. On this basis, the impact of enhancing surface reflective characteristics on point cloud accuracy was verified. The results indicate that altering the reflective properties of an object’s surface can effectively increase the percentage of reflective intensity values obtained by the instrument while maintaining the Lambertian characteristics of the scanned object’s surface, thereby enhancing the overall accuracy of the point cloud data. This outcome can be employed to acquire high-precision point cloud data in applications such as deformation monitoring or precise modeling.k = 1 can accurately predict the measurement errors of the point cloud data.The study also analyzed the raw intensity obtained by the scanner. An error model of ranging was established based on the raw intensity. The experimental results indicate that the model is applicable to laser echoes from non-specular reflection received by the scanner. When the incident angle is 0°, the Lambertian model can be used to determine whether there is specular reflection characteristic. For such data, further experiments need to be set up to analyze the accuracy of point clouds. Furthermore, a single-point error ellipsoid model based on the raw intensity is proposed for assessing the quality of the point cloud. By utilizing a spatial plane model, it has been verified that the error ellipsoid model with In subsequent work, the two analysis methods discussed above will be further refined to develop a ranging accuracy model based on the raw intensity. This model will incorporate segmented functions to represent ranging errors associated with specular reflection and diffuse reflection characteristics. Additionally, in the single point error ellipsoid model, other potential influencing factors will be introduced, such as the impact of the laser spot on the uncertainty of the point cloud.