We describe three mandibular and maxillary oncologic resections performed during 2021 using AR support. In these three patients, we applied a marker-less tracking method based on recognition of the patient’s facial profile. The surgeon, using HoloLens 2 smart glasses, could see the virtual surgical planning superimposed on the patient’s anatomy. We showed that performing osteotomies under AR guidance is feasible and viable, as demonstrated by comparison with osteotomies performed using CAD-CAM cutting guides. This technology has advantages and disadvantages. However, further research is needed to improve the stability and robustness of the marker-less tracking method applied to patient face recognition.In the relevant global context, although virtual reality, augmented reality, and mixed reality have been emerging methodologies for several years, only now have technological and scientific advances made them suitable for revolutionizing clinical care and medical settings through the provision of advanced features and improved healthcare services. Over the past fifteen years, tools and applications using augmented reality (AR) have been designed and tested in the context of various surgical and medical disciplines, including maxillofacial surgery. The purpose of this paper is to show how a marker-less AR guidance system using the Microsoft In the last fifteen years, tools and applications employing augmented reality (AR) have been designed and tested in the context of several surgical and medical disciplines ,3,4,5,6,AR is a technology that superimposes and displays a virtual three-dimensional model on the operator’s field of view. When using AR as a navigation system, it is possible to provide immediate guidance by intuitively displaying useful information for the surgeon, such as anatomical structure in a deep area or virtual surgical planning (VSP).The end user may experience the AR view with various display types, such as standard 2D monitors or tablets, projectors  or wearaNevertheless, as is true of many emerging technologies, no standard method by which AR technology could/should be transferred to clinical practice has yet been developed .Over the past three years, we have employed AR to perform free fibular flap harvesting in a case series of seven consecutive patients undergoing mandibular reconstruction for benign and malignant tumors , and to We also experimentally evaluated the registration accuracy achievable while executing AR-guided maxillofacial surgery tasks, using both commercially available HMD such as HoloLens and “surgery-specific” headsets that are under development ,18,19.The aim of this work is to show how a marker-less AR guidance system using the HoloLens 2 Microsoft HMD can be applied in mandible and maxilla demolition surgery.We describe three mandibular and maxillary oncologic demolitions performed during 2021 using the AR support.In these three patients, we applied a marker-less tracking method based on recognizing the patient’s face profile. The surgeon using HoloLens 2 smart glasses was able to see the VSP superimposed on the patient’s anatomy.A 68-year-old patient with recurrent squamous cell carcinoma of the right maxilla. The patient was studied with a contrast-enhanced CT scan and MRI of head and neck, which showed an uptake of the dye at the level of a neoformation that infiltrated the right maxillary bone and its sinus, and probably extended to the ipsilateral mandibular branch .The study of the neck did not reveal any macroscopically significant lymphadenopathy. The patient reported previous surgery, the first 30 years ago and the second about 20 years ago, for removal of clear cell carcinoma of the maxilla and a recurrence. In addition, the patient had suffered an ischemic stroke 13 years earlier, which had left no consequences. A 56-year-old patient with osteomyelitis of the left jaw. The patient reported the origin of her symptoms following a molar extraction performed 4 years earlier. During this time, the patient underwent first antibiotic therapy and then surgical curettage. Unfortunately, no treatment gave result. The patient was studied with a contrast-enhanced CT scan which showed the extension of the osteitis, and surgery was planned for mandibular disarticulation resection and reconstruction with custom-made titanium plate and free fibula flap. The day before the surgery, the patient refused reconstruction with the fibula, so it was performed only with the plate.A 38-year-old patient also suffering from osteomyelitis of the left mandible following an extraction two years earlier. When usual medical treatment failed, the patient was referred to our surgical unit by his dentist. A CT-scan was performed to study the extent of the disease and to plan surgery. Next, the patient underwent mandibular resection and reconstruction with fibula free flap and CAD-CAM titanium plate.The steps for preparation of the AR guidance application for HoloLens 2 glasses are depicted in Anatomical areas of interest for the surgery were segmented using D2P™ software , a certified software package designed to convert DICOM medical images into 3D digital models. In detail, the maxillary bones, the tumoral mass in the midface region and the facial skin were segmented as separated masks from the CT scan. Then, three-dimensional polygonal surface meshes were generated from each segmented mask and saved in STL (Standard Tessellation Language) format.Using a 3D modeling software  virtual planes for maxillary bone resections were planned following the indications of maxillofacial surgeons, and then converted in STL objects .The obtained virtual models of the anatomical parts and the planned resection planes were imported into Unity 3D , a software provided with a specific development kit for creating augmented reality applications .The tracking algorithm and the registration between the digital models and the real scene were implemented using the Model Target function of Vuforia Engine, which allows the marker-less tracking of a physical object in the real world by recognition of the shape of the 3D object, itself observed from a certain perspective. A Model Target requires that the user holds the AR display at a particular angle relative to the object, and at a particular distance to initialize the tracking. The application typically draws an image (guide view) showing an approximation of the object’s 3D shape from this distance and viewing angle, so that the user just needs to look at the real object while wearing the AR smart glasses until the object matches this guide view. After that, the tracking of the real object and virtual content overlay can begin.(1).At the beginning of the intervention, before skin incision;(2).After skin incision, with the surgical field ready for tumor resection.In this study, the feasibility of AR guidance via HoloLens 2 during maxillary tumor resection was evaluated in two phases of the surgery:For each phase, a specific Model Target was generated to optimize the recognition of a 3D shape in the real scene, and the following virtual-to-real scene registration. During phase 1, the STL model of the whole facial skin was used as Model Target for patient tracking, while during phase 2, a portion of the facial skin, trimmed at the level of the mouth and also including the teeth profile, was used .In both phases, the AR application generates holographic overlays on the real patient, by rendering the underlying anatomical structures , together with the planned virtual planes for resection.Interactable user interface toggles (check boxes) were added to turn off and on the rendering of each virtual anatomical structure  and the planned virtual planes. Voice commands to show/hide the virtual structures were also implemented to provide a completely hands-free AR guidance system. see .A portable high-performance workstation (Intel(R) Core i7-10750H, CPU@ 2.60 GHz, 16 GB RAM, NVIDIA GeForce RTX 2070) was used for the virtual content preparation and for AR development. Then, a UWP  app was built to be installed and run onto the HoloLens 2 smart glasses.The day before surgery, the first surgeon involved in the next day’s intervention was instructed on the use of HoloLens smart glasses and interaction with holograms so that he could feel confident and comfortable with it, and with the developed AR app.During surgery, the mirroring to an external monitor of what the HoloLens sees was set up in order to share the content view of the smart glasses with other surgeons and operating room assistants.Starting from the anatomical 3D reconstruction of the patient’s skull, custom-made cutting guides were designed using 3-Matic software , and then 3D printed. In some cases, the cutting guides were manufactured in Polyamide or in Titanium by direct metal laser sintering like the customized reconstructive titanium plate, in other cases in sterilizable Surgical Guide resin (biocompatible Class I) by stereolithography (SLA) 3D printer .The cutting guides were used in the operating room to help the surgeon to perform the osteotomies, as well as to evaluate the accuracy of the AR application in providing guidance for the osteotomy lines.The surgical accesses used to approach the tumors were those routinely used in maxillofacial surgery.Once the maxillaries were skeletonized, osteotomy lines were made using piezoelectric surgery. Osteotomies were performed according to the VSP visualized during the surgical act. In fact, using voice-controlled HoloLens, it was possible to superimpose the surgical planning on the patient’s anatomy. Thus, the surgeon was able to draw the osteotomy lines while simultaneously looking at both the patient and the planning, without the need to look away from the surgical field to look at a navigator monitor .Once the osteotomy lines were drawn, the custom cutting guides were used for treating the patient. The cutting guides were useful also to evaluate any error in the previously drawn corticotomy line under AR guidance.Below we detail the surgical procedure of the first patient treated with this methodology.A Weber–Ferguson surgical incision was performed under general anesthesia. The maxilla was skeletonized to expose the anterior wall of the maxillary sinus and approach the tumor anteriorly. A lip-split mandibulotomy was performed to better approach the tumor posteriorly at the level of the pterygoid laminae.Before performing the osteotomy lines of the planned type IIIb maxillectomy according to Brown’s classification , the surThe surgeon traced the corticotomy lines by following the VSP superimposed on the patient’s anatomy through the AR glasses. Using voice commands, it was easy for the surgeon to visualize the surgical plan superimposed on the defect. The use of HoloLens HMD allowed a three-dimensional view of the tumor and a better approach to it . Next, tThe corticotomy and osteotomy lines drawn for each spatial plane were compared and measured.To evaluate the distance between the corticotomy line traced using AR and the osteotomy line provided by the custom-made cutting guide, we used a caliper during surgery. Moreover, some screenshots of the operative field were taken using the HoloLens camera when both the corticotomy line was projected in AR and the cutting guide was positioned on the patient. Then, post-operatively, the distance between the two lines was estimated analyzing the recorded intraoperative screenshots .A functional latero-cervical lymph node emptying was performed according to the guidelines.Next, the maxillary defect was reconstructed with an antero-lateral thigh flap. The vascular pedicle was anastomosed to the superior thyroid artery and the thyro-lingual trunk. A titanium mesh was used to reconstruct the orbital floor.Three patients were treated consecutively with the presented AR guidance approach. They were two males and one female with a mean age of 54 years  with recurrent oral squamous carcinoma (SCO) of the right maxilla, and two osteomyelitis of the left mandible.All surgeries were performed without complications and in line with surgical times for the same pathologies.There was a total of seven groups of osteotomy planes, each composed of corticotomies performed using AR and the corresponding “real” planes provided by the custom-made cutting guides. Four corticotomy lines and four osteotomy lines were drawn for the first patient, and two for each plane of maxillectomy, representing four groups; One corticotomy and one osteotomy line were drawn for the anterior mandibular resection plane for the second patient, representing one group. Lastly, for the third patient, two corticotomy and two osteotomy lines were drawn, two for each mandibular resection plane, for a total of two groups of osteotomy planes.For all cases, comparison of the corticotomy lines drawn with the aid of the AR and the osteotomy lines drawn with the use of cutting guides showed a discrepancy under 2 mm.Following debulking surgery, two patients were simultaneously reconstructed with microvascular-free flaps in accordance with VSP.Surgery remains the gold standard for the management of maxillofacial neoplasms . CT and An in-depth understanding of the size of the tumor and its relationship with adjacent structures allows resective surgery to be performed with clear margins and without unnecessarily harming anatomical boundaries.Nowadays, CAD-CAM cutting guides allow the VSP  to be brThe application of intraoperative navigation has represented a valid alternative, effective in terms of cost/benefit balance for the translation of the VSP into operative reality, and with good results in terms of precision and safety ,26; howeMoreover, current intraoperative navigation systems allow surgeons to inspect and interact with 3D images and objects on a flat-panel display , thus liInterferences  during surgical procedures can be deleterious .With holographic imaging technology, it is now possible for users wearing head-mounted displays to manipulate and interact with virtual objects in real time .In this study, HoloLens 2 smart glasses was used as a head-mounted display to project holograms onto the patient’s anatomy. During the various steps of the surgery, the STL model of the whole facial skin was first used as a target model for patient tracking, followed by a portion of the facial skin cut at the level of the mouth and including the profile of the teeth. Throughout the surgery, the AR application generated holographic overlays on the real patient, rendering the jaw and tumor together with the virtual planes planned for resection. This determined several advantages: visualization of the tumor, the critical surrounding structures and the virtual surgical plan intraoperatively, all consistent with the real patient’s anatomy; display of the planned osteotomy line in real time and the ability to follow it without looking away from the surgical table; and manipulation of the augmented digital content using the voice control function supported by the HoloLens HMD which leaves the surgeon’s hands free to execute the surgical task.In previous studies on mixed reality, the superimposition method was used to overlay holograms over actual anatomical structures, and manual matching was then performed ,31. The To the best of our knowledge, the application of augmented reality using the patient’s face directly as a model target for patient tracking has not been reported previously in oral and maxillofacial surgery. This allows surgeons to superimpose the holograms of the VSP directly onto the patient’s face without using any marker-based tracking system, thus freeing the surgical field from cumbersome objects that may obstruct visual and manual tasks, or that must always be in the line of sight of an external camera. This provides more surgical room and manual dexterity.In contrast, a major problem reported in the literature with the use of HoloLens is visual discrepancy. The hologram site may appear in a different spatial position in the assistant’s view, even after registration by the surgeon. Galati et al. reported a discrepancy of 4.5 cm when the same reference point was viewed from different perspectives. Visual discrepancy is potentially dangerous .https://www.vostars.eu/, accessed on 6 November 2022), which has already demonstrated on a phantom very promising submillimetric accuracy in guiding high-precision surgical tasks [® HoloLens 2, it is possible to look at the real environment around you, as it is an “optical see-through (OST)” head-mounted display (HMD); i.e., it uses semi-transparent surfaces of projection that preserve the user’s direct view of the world. This aspect confers a clear advantage, particularly when used to interact with objects in the peripersonal space, as it allows the user to maintain an unaltered and almost natural visual experience of the surrounding world.Another problem that has been reported is that the operating light may affect the quality of the hologram, and that the overlap with the surgical field may obstruct the view of anatomical structures . As preval tasks ,19,34. MThe aim of this study was to evaluate the feasibility of performing a resective maxillary osteotomies using augmented reality.The use of augmented reality appears to be safe and effective for tumor resection in the oral and maxillofacial region. This technology has advantages and disadvantages. However, further research that aims to improve the stability and robustness of the marker-less tracking method applied to patient face recognition is needed.


Uneven terrain walking is hard to achieve for most child-size humanoid robots, as they are unable to accurately detect ground conditions. In order to reduce the demand for ground detection accuracy, a walking control framework based on centroidal momentum allocation is studied in this paper, enabling a child-size humanoid robot to walk on uneven terrain without using ground flatness information. The control framework consists of three controllers: momentum decreasing controller, posture controller, admittance controller. First, the momentum decreasing controller is used to quickly stabilize the robot after disturbance. Then, the posture controller restores the robot posture to adapt to the unknown terrain. Finally, the admittance controller aims to decrease contact impact and adapt the robot to the terrain. Note that the robot uses a mems-based inertial measurement unit (IMU) and joint position encoders to calculate centroidal momentum and use force-sensitive resistors (FSR) on the robot foot to perform admittance control. None of these is a high-cost component. Experiments are conducted to test the proposed framework, including standing posture balancing, structured non-flat ground walking, and soft uneven terrain walking, with a speed of 2.8 s per step, showing the effectiveness of the momentum allocation method. A biped robot has a high degree of freedom and is expected to achieve stable motion in complex environments. Thus, biped robot motion control is one of the research hotspots in the field of robots. To enable stable walking of a humanoid robot, various zero moment point (ZMP) controllers are designed to track the motion trajectory and resist external force disturbance when the robot is walking ,2,3,4. WWhen ZMP controllers are to change the contact force between the robot and the environment, some other controllers focus on changing the contact position of the robot to further improve the anti-interference ability. The capture point control uses the instantaneous capture point to calculate the landing position ,7 of theHowever, for child-size biped robots, because of the inability of their joints to perform precise force control, it is difficult for them to precisely control the position of ZMP in the supporting polygon. Moreover, when the ZMP is out of the support polygon, for example when the robot is disturbed by a large external force, these ZMP controllers have little effect on the balance control process. As for contact-position-related methods, the robot needs to quickly adjust the landing position during the stepping process, so the joints needs to have a fast response ability to action commands, which may exceed the performance of most child-size biped robot joints.Apart from contact force control and contact point control, the balance control of humans and other bipeds also involves the coordinated movement of the upper body. Researchers have also explored balance control methods based on the upper body  and centHe et al. proposed a method to compensate static walking trajectory with angular momentum offline to improve the dynamic characteristics of the whole body trajectory . In this•Providing a momentum allocation method considering the change of centroidal angular and linear momentum during robot movement to increase the stability of robot locomotion.•Demonstrating the effectiveness of angular momentum compensation and linear momentum compensation.•Realizing uneven terrain walking in a child-size humanoid Briefly, the main contributions of this paper are as follows:The rest of this paper describes our approach based on the following sections. In ZMP is a widely used ground reference point for biped planning and control. However, when considering the dynamics of multi-rigid bodies, it cannot accurately calculate the trajectory of the robot’s center of mass (CoM) ,14. As wx-direction.Popovic et al. discussed the difference between ZMP and CMP  and the rection.xZMP=xCoMThe CMP ground reference point was introduced in  to discoThe relationship between CoM and ZMP is determined by Equation , ignorinOrin et al. proposed a centroid dynamics model . Using tAccording to Newton’s laws of motion, the rate of change of angular and linear momentum at the CoM, l forces ,24. FiguConsider the case where only continuous contact wrench acts on the robot. Express the relation between the momentum rate of change and external forces in the sagittal plane  in each sole, shown in jth FSR unit, Using four FSR units, the ZMP of the sole can be calculated as follow:M, B, and K are the designed control parameters, which are obtained by empirical tuning in this work, and After obtaining the ZMP, we can apply admittance control using the following controller. Both the x-axis and y-axis can use the same control law.Due to the low measurement accuracy of the FSR unit, this controller may lead to trajectory divergence when the robot is in the single support phase. So we only enable the controller for the double support phase.Due to the limited width of the robot’s sole in the Y direction, it is more difficult to maintain balance and track the CoM trajectory in the Y direction than in the X direction. In order to enhance the robot’s ability to maintain Y-direction balance, in addition to the above momentum allocation and attitude controller, we also use arm swing to adjust the position of the robot’s CoM for balance control. We use the arm swing control to adjust the CoM position rather than directly changing the trunk position for CoM adjustment because we found that adjusting the trunk position in the Y direction to achieve centroid position adjustment will bring additional unstable factors and make it easier to fall. We speculate that adjusting the centroid by changing the trunk position involves changing whole-body momentum and foot CoP, which has a negative impact on the momentum allocation and attitude control.In the arm swing controller, we use the trunk inclination and angular velocity as inputs and use the PD controller to calculate the compensation angle of the arm swing, which is similar to the posture controller in Equation . Due to Note that the arm swing control mainly acts on the coronal plane, while the momentum allocation control mainly acts on the sagittal plane. The left and right swing of the arm has little impact on the momentum and angular momentum in the X direction of the robot, so the two control methods can coexist without affecting each other.The overall control diagram is shown in In order to test the effectiveness of the momentum allocation control framework, we carried out three experiments: standing balancing, discrete terrain walking, and uneven terrain walking See . Table 2We tested the disturbance resistance of the robot in a standing state using the angular momentum compensating linear momentum method. In the simulation, a humanoid virtual prototype with a 60 kg weight and a 1.3 m height was pushed by a 400 N force lasting for 0.1 s and recovering in 2.0 s. In the hardware test, the robot was hit by a volleyball with speeds ranging from 6.5 m/s to 5.3 m/s in multiple tests. We can see from The above experiments show the effectiveness of the momentum allocation, but they also show that the momentum allocation based on angular momentum is mainly used to speed up the stable process of the robot after being pushed. However, it has a limited effect on reducing the trajectory deviation of the robot.Different from the standing balancing experiment, in the discrete terrain walking experiment, we found that the swing of the trunk easily causes instability. In this case, we manually tuned and weakened the feedback coefficient of the angular momentum compensation strategy so that linear momentum compensation strategy plays a major role.We tested the discrete terrain walking control with two rows of circular support columns, as shown in We recorded the convergence time of each controller in a walking process, and obtained the histogram shown in When walking on uneven ground, due to the unknown terrain, the robot may have a large landing impact. When the robot steps on a high obstacle when moving forward, it causes the robot to fall backward with clockwise angular momentum, as shown in We experimented with structured uneven terrain walking and soft-ground walking. The step time cycle was about 2.0 to 3.5 s per step, which was limited by the controller convergence time and was different in every step. The step length was about 8 cm, which was limited by the kinematic constraint and walking stability and was determined empirically. The obstacle height was about 2 cm, approximately 3.3% of the total body height of 65 cm. The video frame capture of walking on structured uneven ground is shown in Based on the proposed method, three experiments were carried out: standing balancing, discrete terrain walking, and uneven terrain walking. In the standing balancing simulation, a humanoid can recover from an impact of 400 N/0.1 s. In the hardware experiment, a volleyball was used to hit the standing robot, and its anti-disturbance control ability with angular momentum compensation control was observed.In the discrete terrain walking experiment, the robot was placed in front of multiple support columns and needed to walk through them. Although the foot of the robot is not small relative to its height, the contact area between the foot and the support column was limited. The linear momentum compensation method was implemented in the double support phase, improving the walking success rate from 40% to about 90%.Finally, in the uneven terrain walking was carried out to test the effectiveness of the whole control framework. The robot can walk on the rugged ground with an obstacle height of about 2 cm, about 3.3% of the total body height of 65 cm. The average walking speed was 2.8 s per step, with a step length of about 8 cm. The effectiveness of the momentum allocation control was verified.As a comparison of different humanoid prototypes,  shows thFrom the above experiments, it can be seen that the proposed method can achieve anti-disturbance control of child-size humanoid robots and can improve the stability of walking on non-flat terrain. However, it is also found that it is difficult to improve the walking speed with the proposed method. Based on the experimental tuning process, we speculate that the possible factors limiting walking speed are: (1) the IMU sensor’s measurement bias, (2) gear backlash of joints, (3) the robot’s lower control frequency (100 Hz), and (4) the position tracking accuracy of joints. Moreover, there are three controllers in the framework, and there may be interference between them. However, in the uneven terrain walking experiment, all three controllers were necessary to prevent the robot from falling, and an experimental process that can accurately compare the effects of the three controllers is still being designed.This paper studies the balance control problem when the ZMP moves to the edge of the supporting polygon, and the landing position cannot be changed. This situation often occurs in the process of complex motion of robots. Solving this problem can greatly improve the motion control effect of robots. Among the three strategies of balance control, walking strategy, hip strategy, and ankle strategy, hip strategy is the most suitable for this situation. This paper designs a simple momentum allocation method based on a PD controller, which is combined with a whole-body posture controller and a landing admittance controller to implement robot momentum allocation control. Based on the control framework, this paper verifies the effectiveness of angular momentum compensation and linear momentum compensation methods, respectively. The control framework enhances the anti-disturbance ability of the robot when standing and improves the stability in the X direction when walking on discrete and uneven ground. The above experiments verify the effectiveness of momentum allocation on the physical prototype.This paper also has the following limitations. Firstly, there is the limitation of hardware. The limited trajectory tracking performance of the robot makes the stable convergence process slow, which leads to a slow walking speed of 2.8 s per step, with a step length of about 8 cm. Secondly, the principle of momentum allocation in this paper is to reduce the momentum and angular momentum to zero at the same time in the stable control stage of walking. This is suitable for control tasks such as intermittent walking, which has zero angular momentum. However, other control tasks such as jumping and dancing will involve non-zero angular momentum, and a more general method of momentum allocation is needed.As for future work, we will first focus on improving the hardware experimental platform and design more detailed controller verification steps to explore the performance of the three controllers. Other research directions include the design of momentum allocation methods with non-zero angular momentum trajectory such as continuous walking, running, and dancing. It is also an interesting topic to use quadratic programming (QP) and other methods to map the allocated momentum to whole body trajectory to achieve more accurate momentum control.


This paper aims to present the unique, to the best of our knowledge, case of entrapment of a standard vacuum drainage tube in the articulating surfaces of the cup of dual-mobility total hip arthroplasty. A 75-year-old woman with end-stage idiopathic avascular necrosis of the left femoral head was referred to the arthroplasty service of our tertiary orthopedic department. She underwent a scheduled and uneventful total hip arthroplasty with a press-fit dual-mobility prosthesis through a standard posterior approach. On the second postoperative day, the attempt to remove the standard vacuum drainage was unsuccessful. Consequently, the patient underwent urgent re-operation. The drain tube was found entrapped between the articulating surfaces of the posterior-inferior aspect of the dual-mobility cup and was uneventfully removed. The patient was discharged with no further events three days after her second operation. Our unique rare case increases awareness when performing even routine everyday surgical procedures because a rare complication may occur irrespective of the level of vigilance of the surgeon and can potentially compromise the outcomes of an otherwise well-performed operation. Total hip arthroplasty is considered to be one of the most successful orthopedic procedures . Since iWe present the unique, to the best of our knowledge, case of postoperative entrapment of a standard vacuum drainage tube in the articulating surfaces of the cup of a dual-mobility cementless total hip arthroplasty, which necessitated a re-operation for its removal.A 75-year-old Caucasian woman with intractable pain in her left hip and subsequent inability to walk and perform everyday activities was referred by her general practitioner to the arthroplasty service of our tertiary orthopedic department. Her significant medical history consisted of type II diabetes mellitus, dyslipidemia, arterial hypertension, acromegaly, and mitral valve insufficiency under medication. Currently in complete remission, she had also undergone trans-anal resection of a colon carcinoma more than a decade ago.Her symptoms  started 18 months before her referral and gradually deteriorated. Upon her initial evaluation, all her hip movements were painful and she was using two crutches for walking, being ambulatory only within her household limits. Passive hip flexion was 60°, there was a lack of extension of approximately 10°, and internal and external rotation movements were not possible. A standard radiograph revealed the existence of a Ficat  end-stagThe patient was offered a total hip arthroplasty to her left hip, which she accepted. She was operated on two months later. Under general endotracheal anesthesia and through a posterior approach, a press-fit dual-mobility total hip arthroplasty was implanted . Following the insertion of the implant and the reduction of the hip joint, and after full and unrestricted movements toward all directions were checked and secured, as a standard operative procedure, a 14 Redon vacuum drainage tube was inserted and placed near the articulation. After the completion of the operation, a standard hip and pelvis radiograph was taken Figure .On the second postoperative day, the Redon canister was filled with 160 ccs of blood and it was decided to be removed. However, the attempt to remove it was unsuccessful. Consequently, the patient underwent urgent re-operation. The drain tube was found entrapped between the articulating surfaces of the posterior-inferior aspect of the dual-mobility cup. It was removed by gently yet firmly applying extraction force while the hip was slightly flexed, adducted, and internally rotated. The removed tube was checked and was found to be compressed at a distance of 4 mm from its tip, without any sign of a rupture. No evidence of a remaining part of the tube could be found, and full and unrestricted movements of the hip joint were confirmed. The wound was thoroughly irrigated with saline and sutured in layers. The standard perioperative antibiotic protocol for patients undergoing total hip arthroplasty (three doses of 2 g of cefazolin during 24 hours) was used for the re-operation. The patient was released with no further events and on schedule three days following her second operation. Upon her latest follow-up visit six months postoperatively, clinically, she was capable of full weight-bearing, she was able to walk without any assistance and/or aids, she reported no problem, and she was fully satisfied with the result of the operation. The radiograph taken on the same day confirmed the proper positioning of the prosthesis Figure .Total hip arthroplasty has been a milestone in the treatment of patients with hip osteoarthritis and in orthopedic surgery in general. Since the introduction of modern total hip arthroplasty by Sir John Charnley, it has continuously evolved . New matEvery complication or malfunction of the components of total hip arthroplasty and/or a procedure related to the operative process itself, necessitating a re-operation, increases the possibility of an infection, with potentially catastrophic results . This isDespite thorough literature research, no data could be found regarding the incidence of vacuum drainage tube entrapment following total hip arthroplasty. This is certainly an infrequent and relatively “benign,” yet familiar to all surgeons, complication. The drainage tube is usually accidentally sutured during the closure procedure, which necessitates re-operating on the patient to be released. To the best of our knowledge, the entrapment of the drainage tube within the articulating surfaces of a dual-mobility total hip arthroplasty cup has never been reported in the literature. We postulate that this must have occurred either during the transfer of the patient from the operating table or during the immediate postoperative period and gradual mobilization because upon completion of the operation it was not evident. To prevent such complications, we recommend shortening the drain to keep it at a distance from the dual mobility to avoid entrapment during mobilization.The limitation of our case report is the lack of a transoperative photograph documenting the entrapment of the drainage tube within the articulating surfaces of the dual-mobility cup. Nevertheless, our rare case aims to increase awareness when performing even routine everyday surgical procedures because a rare complication may occur irrespective of the level of vigilance of the surgeon and potentially compromise the outcomes of an otherwise well-performed operation.Every complication following total hip arthroplasty, which necessitates a re-operation, increases the possibility of an infection, with potentially catastrophic results. This is especially true when complications occur during the immediate postoperative period. This rare case proves that non-expected complications may occur even to experienced surgeons performing operations that they are familiar with. Extreme vigilance maintained during the whole operation and attention to every little detail (even the positioning and length of the drainage tube) are needed to secure the best outcomes.


An anterior inferior cerebellar artery (AICA) that crosses the right trigeminal nerve is an uncommon arterial anatomic variation. In this anatomical position, it is difficult to separate or move the offending blood vessels and nerves. We report an uncommon case of trigeminal neuralgia (TN) caused by compression of the trigeminal ganglion by a branch of the AICA.A 34-year-old man with 5 years history who complained of pain on the right side of the face (area V1). The symptoms gradually worsened, and the pain episodes became intense and frequent. Magnetic resonance imaging (MRI) of the cerebrum showed a small blood vessel passing through the right trigeminal nerve. Microvascular decompression (MVD) was performed,because medication was ineffective. Intraoperative exploration confirmed that the vessel which was a branch of the AICA passing through the right trigeminal nerve. As while the artery was temporarily clipped, electrophysiological monitoring showed a decrease in the amplitude of nerve activity. As the artery was considered too important to be sacrificed, the space between the nerves was enlarged mildly, the artery was liberated, the Teflon implant was shredded and placed between the artery branches and nerve to make the blood vessels as perpendicular as possible to the nerve. The patient had no neurological dysfunction and no pain after 8 months of follow-up.MVD is an effective treatment for artery-induced trigeminal nerve compression, but we report a novel procedure that avoids the complication of facial numbness caused by cutting the offending vessels and incision of the trigeminal nerve. Vascular compression of the fifth cranial nerve in the cerebellopontine angle commonly as the cause of TN have drawn the attention of neurosurgeons. However, sometimes TN is also be induced by idiopathy or lesions including cysts , tumors Based on the consensus of vascular compression of the fifth cranial nerve can caused TN.MVD has been generally considered the standard microsurgical treatment for TN. The superior cerebellar artery (SCA) is the most common causative vessel, and other vessels involved either alone or in conjunction with other vessels were the AICA, the vertebral artery, and the petrosal vein. A transfixing artery has rarely been reported in the literatureas the offending vessel in TN . The tasThe authors report a patient with TN caused by a penetrating artery; the patient was successfully treated by microvascular decompression. In addition, the authors reviewed the relevant literature.A 34-year-old male had a 5-year history of facial pain in the right V1 regions. In 2016, oral carbamazepine uptake resulted in pain relief. In March 2021, the pain episodes became intense and frequent, increase the dose of carbamazepine was ineffective in relieving the pain. Notably, the patient did not have classic “trigger point pain”. Each time, the patient had a pain episode lasting 5 to 8 min, which was significantly longer than the duration of classic symptoms. In September 2021, the patient was admitted to our institute for MVD. The patient underwent high-resolution three-dimensional fast imaging employing steady-state acquisition magnetic resonance imaging (3D FIESTA MRI). The 3D FIESTA sequence can clearly visualization the neurovascular anatomy of TN patients. 3D FIESTA imaging which has high contrast and spatial resolution could depicted structures surrounded by cerebrospinal fluid (CSF). On images, There are depicted as different signal-intensity structures between blood vessels and the nerve and the CSF.MRI of the trigeminal nerves revealed a small vessel penetrating the right trigeminal nerve. Multimodal MRIs were then coregistered with open- source software 3D Slicer, and 3D image reconstruction was performed to generate virtual reality (VR) images for detecting possible penetrating arteries (Fig. A regular retrosigmoid suboccipital craniotomy was carried out to access the cerebellopontine angle, and we confirmed that the blood vessel transfixing the right trigeminal nerve was a branch of the AICA. Furthermore, the whole course of the trigeminal nerve was carefully examined, and no other offending vessels were founded Fig. A.Fig. 2AWe attempted to cut the offending artery and separate it from the nerve. After 30 min of temporary occlusion of the offending artery, electrophysiological monitoring showedthat the latency of the V wave exceeded 1 millisecond, and each waveform decreased by more than 50%, indicating that the hearing pathway was damaged and that the patient was at risk of hearing loss (Fig. The trigeminal nerve was inspected carefully, the space between the nerves was mildly enlarged, and the artery was liberated. A shredded Teflon implant was placed between the artery and nerve branches to make the blood vessels as perpendicular as possible to the nerve. Importantly,the shredded Teflon implant was placed in the crossing space between the offending artery and nerve rather than at the passing point Fig. B. The paMVD was first used by Gardner and Sava as a surgical treatment for TN, and it was later popularized by Peter Jannetta . It has In rare cases, arteries or veins may pass through the trigeminal nerve. The passage of blood vessels through the trigeminal nerve is an uncommon cause of TN. Because of this positional relationship, it is difficult to separate or move the offending vessel and nerve. This type of compression is difficult to treat properly. Patrick  reportedTashiro et al.reported the first three cases of arterial penetration into the trigeminal nerve . Two of Helbig divides TN into two categories, type I: The blood vessel crosses the combined sensory and motor trigeminal nerve branches between motion and sensation; type II: The blood vessel crosses the sensory branches of the trigeminal nerve . Helbig Up to now, there is still no standard therapeutic schedule for this situation. JunyaJito and Kazuhiko Nozaki used the slinging method to shift blood vessels and nerves . SlinginAccording to the existing literature, veins with a small diameter can simply be cut off. However, coagulating or cutting off a vein with a sufficiently large diameter may lead to facial numbness and corneal sensation. If the penetrating vessel is an artery, it is difficult to move the vessel for decompression. At present, the primary method is to allow free movement of the artery. The nerve root is cut to shift the artery into the pad. Alternatively, the blood vessels are moved to the distal end of the intracranial segment of the nerve root by slinging to the bone.We communicated with the patient before the operation. The young patient indicated that he would not accept facial numbness. We observed that the artery was very small, and we attempted to sever it. The operator clipped the artery, and the amplitude of nerve activity decreased 20 min later. Immediately after the clip was removed, the amplitude returned to normal. When blocking the aneurysm, we found it impossible to cut the artery. Because cutting the nerve root is bound to cause facial numbness, we abandoned the method of cutting part of the trigeminal nerve. When inserting shredded Teflon, we were cautious of the risk of facial sensory impairment. Finally, we opened a slight gap around the vessel and used shredded Teflon to pad the space between the vessel and nerve so that these structures could maintain as little contact as possible. Postoperatively, thepatient’s pain was relieved, and no facial numbness remained. There was no recurrence after 8 months of follow-up.We describe an uncommon case of TN caused by an artery penetrating the trigeminal nerve. However, the young male patient made clear to us that facial numbness was unacceptable. To satisfy the patient’s request, we merely separated the nerve and artery and used shredded Teflon placed between the nerve and the culprit vessel to position the artery as perpendicular as possible to the nerve, which decreases the contact area between the artery and the nerve.The main advantage of this procedure is that it does not entail cutting the offending vessels and incising the trigeminal nerve in the longitudinal or transverse direction, which can lead to facial numbness. This procedure may be the optimal choice for young patients who cannot accept facial numbness.


Pedestrian detection (PD) systems capable of locating pedestrians over large distances and locating them faster are needed in Pedestrian Collision Prediction (PCP) systems to increase the decision-making distance. This paper proposes a performance-optimized FPGA implementation of a HOG-SVM-based PD system with support for image pyramids and detection windows of different sizes to locate near and far pedestrians. This work proposes a hardware architecture that can process one pixel per clock cycle by exploring data and temporal parallelism using techniques such as pipeline and spatial division of data between parallel processing units. The proposed architecture for the PD module was validated in FPGA and integrated with the stereo semi-global matching (SGM) module, also prototyped in FPGA. Processing two windows of different dimensions permitted a reduction in miss rate of at least 6% compared to a uniquely sized window detector. The performances achieved by the PD system and the PCP system in HD resolution were 100 and 66.2 frames per second (FPS), respectively. The performance improvement achieved by the PCP system with the addition of our PD module permitted an increase in decision-making distance of 3.3 m compared to a PCP system that processes at 30 FPS. Pedestrians, called vulnerable road users (VRUs), represent more than half of all the global deaths in transit accidents . As carsPedestrian detection (PD) is a fundamental and critical component in PCP systems ,4. SinceOne way to work around the deficiency of detecting distant pedestrians is to process data with increasing resolutions . Image sUnfortunately, the existing detectors based on convolutional neural networks (CNN) have a high computational cost that prevents us from obtaining efficient processing solutions ,15. For ng rates  due to tDue to the excellent parallelism capabilities of the field-programmable gate array (FPGA) platforms ,17,18, sIn  the authIn  the authIn addition to the image pyramid, another approach to increasing the ability to locate pedestrians at different distances involves processing detection windows of different dimensions. While the pyramid aims to encounter pedestrians larger than the detection window dimension, the windows of various sizes permit locating pedestrians that appear with smaller dimensions. Our previous work  demonstrThus, this work proposes an FPGA-based processing architecture of our detector  to furthThe proposed architecture comprises several parallelism strategies at the frame level and detection windows level to performance optimization. At the frame level, modules are instantiated to process detection windows of equal and different dimensions parallel at each pyramid level. At the detection windows level, the most significant contribution of this work, a set of smaller SVM units is created to process a bunch of different overlapping detection windows in parallel to ensure the completion of the current row before the arrival of the next row. These strategies guarantee a processing throughput of one pixel per cycle, achieving the same performance as the input camera without frame loss or pixel input interruption.In addition to the PD system architecture, another essential contribution to the performance improvement of the PCP system is in the pedestrian distance estimation step that depends on the results of the Semi-Global Matching (SGM) . Since tA heterogeneous platform based on a general-purpose processor (GPP) and FPGA implements the proposed PCP system. The GPP processes the less computationally expensive steps. The adopted platform is the Intel HARP version 2 (HARPv2) . The impThe impact of the proposed PD system on the collision prediction task is also evaluated. Synthetic collision scenarios were created involving an occluded pedestrian crossing in front of the moving car to carry out this assessment. The processing rate of the entire PCP system for processing HD images is approximately 66.2 FPS. This rate allowed an increase of 3.3 m in decision-making distance for a vehicle traveling at 60km/h compared to our previous PCP system [A highly parallel architecture with performance optimization of the HOG-SVM-based detector with support for detection windows of different dimensions and the image pyramid.A strategy involving several SVM units to parallel processing of a massive amount of detection windows.Performance optimization of the PCP system by the integration of the proposed detector and the SGM module.In summary, the contributions of this work are pointed out as follows:This paper is organized as follows: This section describes the techniques and algorithms of the pedestrian detection (PD) system. It is essential to make it clear that this section is not the contribution of this work but of our previous work . It servThe (PD) system steps are shown in The detector starts converting each pixel from the RGB to the grayscale color space by weighted sum per channel. The image pyramid step  produces several re-scaled images of smaller dimensions than the original image. At level 1 of the pyramid is the original image, while at the other levels, the images are re-scaled through resizing. The scale factor The HOG and SVM pyramid steps  apply the HOG and SVM techniques to each image generated by the image pyramid. Each The HOG and SVM processing steps are shown in Experimentally, it was possible to observe that the HOG-SVM based detectors can only detect pedestrians whose image dimensions fit close to the limits of the detection window dimension. The image pyramid is one of the approaches to adjusting the dimension of larger pedestrians. Pedestrians that are much smaller than the detection window are not detectable due to the property of the HOG and SVM combination that cannot capture pedestrians in high pose variability. Although the samples of various sizes are added in the learning phase, the SVM classifier does not converge to a model that captures all these variabilities.On the other hand, smaller detection windows permit to detection of smaller pedestrians that are usually farther away from the vehicle because these pedestrians fit better on these windows. The image pyramid helps increase detection capability, but the increase in the pyramid depth above a limit provokes the detector to introduce too many false positives, besides not being able to find larger pedestrians. This is because smaller detection windows contain less edge information needed to classify pedestrians.The solution proposed in  is to haAnother aspect of this system is the data processing flow at each step. The execution can follow any flow at the software level once the data is in memory. However, on hardware systems, the execution follows the left-to-right and top-to-bottom streaming flow similar to pixel scanning of traditional hardware cameras.The resize, HOG, and SVM steps are detailed in the following sections.This step resizes the grayscale source image to a target dimension using the bilinear interpolation technique . This teEquation .(1)xsrcThe constants Equation  as μl, μFinally, the intensity Equation .(3)IdstThe resized image is obtained by calculating all pixels within the new dimensions of the rescaled image, using Equations –3)..3).In this step, the necessary features are calculated for the classification. The HOG method comprises three main steps: gradient, cell, and block.This step calculates the magnitude and the orientation angle of the gradient of each pixel from the input image. By using Dalal’s formulation in , the 1-DEquation .(4)dx(xFrom the pair Equation .(5)mIt is important to highlight that the neighbor bin index The Equation  guaranteThe cell histograms are grouped into blocks and normalized within the block to increase immunity to lighting variations, as shown in refer to .x and y, respectively. This parameter must be a multiple of the HOG cell dimension.This step performs the scan using sliding window techniques on the HOG features image. In each detection window, the existence, or lack thereof, of a pedestrian is checked. Detection windows of a given In each detection window, the descriptors are formed by concatenating the HOG features into a single one-dimensional vector. Then, the system verifies the existence of a pedestrian through the SVM linear classifier that compares these descriptors with a reference model produced by a supervised learning phase. Since it is a binary classifier, SVM linear will return the result whether or not the window contains a pedestrian. Therefore, it is crucial to guarantee that the HOG features composition order in the one-dimensional vector in the test phase is the same as in the SVM training phase. Otherwise, the classification will not work correctly.In mathematical terms, a detection window is indicated by b are, respectively, the weight and bias of the linear SVM, defined through the supervised learning phase. The result The parameters The function Equation  is used The constant This section describes the hardware architecture of the PD system, detailing the modules that implement each step of the system described in A fully parallel processing strategy is required to support image pyramid and different size detection windows strategy without frame loss. The Image-P, HOG-P, and SVM-P modules instantiate, respectively, resize, HOG, and SVM modules to process each image in parallel according to the The system receives pixels in the RGB color space and returns the bounding boxes (B.B.) along with their respective confidence. The signal Found Pedestrian has value 1 when the detection window is classified as having a pedestrian , and 0 All modules follow the pixel scan flow of most hardware cameras, from left to right and top to bottom. Furthermore, the PD system works in a slave mode, always waiting for the external source to inform that it has a pixel available. The PD system supports one pixel per clock cycle. Whenever the available signal is at logic level 1, an input pixel is available, and the PD system readily reads the three channels of that pixel and processes them. This slave approach is only possible because all modules can operate without interrupting the input flow of pixels. This feature allows the PD system to easily integrate through digital pins in hardware cameras that use a serial protocol .In addition, the entire system operates as a continuous flow of processing. Whenever any module has data ready in the input, it will process it and store the partial results. If any response data can already be provided, the module will signal for the next module to receive this data and process it.All modules of this system operate with numbers in fixed-point representation. All arithmetic operations come from integer operators adapted for fixed-point. For simplicity, bits wide of the fractional part, defined by All modules process data in the pipeline with a minimum of operations per stage. Any mathematical operation module  will always have pipeline cycles quantity equal to the operated data width. Consequently, the modules have more pipeline stages, but on the other hand, the PD system achieves higher processing frequencies. After filling the pipeline stages with data, the system processes one detection window per clock cycle with no frame loss.Next, the hardware modules that implement each step in the proposed PD system are detailed.An RGB pixel has three channels, in which the intensity of each channel has The resize module will process the new image converted to grayscale. The module has as parameters the resolutions of the input image and the output image defined from the The processing follows the Equations –3). The. The3). Two critical aspects can be noted in these Equations. The first aspect is that if quations  and 2)..Spyr is The first aspect is solved by implementing a strategy of multiple interpolation units. These units process Equation  at interq. When the input image pixel referring to the first calculation of q is available, it is multiplied by the weights q, is available , it is multiplied by Each interpolation unit has a Mapping Checking module that verifies which input pixels will participate in calculating the new resized pixels. Consider the calculation of a given pixel Equation . When thThe second aspect is solved by noting that the scale x axis does not vary.The resize does not store this mapping and weight data for the entire image, as it would not be a scalable solution concerning image resolution. Instead, the Mapping Calculator module defines a whole line of mappings Equation , split, The mapping The HOG module includes three submodules following the steps detailed in As the resized pixels are ready, this module first calculates the gradient components Equation . This caEquation . The sliAs the components Equation . The firThe second component peration . Thus, aAs the gradient module and orientation are available, the cell module calculates the cell histograms. The processing architecture is shown in Equation , performFollowing Equation , the unsEquation . InsteadThen, as part of the From the value of the gradient index From the result of each cell, the block processing module performs the normalization of each bin within a block through Equation . The arcEach bin goes to the Denominator module, which calculates the denominator from Equation . First, It is important to note that the one-bin-at-a-time processing approach is only possible if the number of cycles required to process a line of blocks is less than the number of cycles waiting for the following line of blocks. Otherwise, some strategy to lock reading or buffering would be necessary. This feature exists because, between one block row and the next, there are As the blocks  are available, the SVM module calculates the confidence Equation  and thenEquation .Overlapping between windows is a challenging aspect of achieving streaming processing. Since the stride in pixels between neighboring windows is smaller than the detection window dimension, multiple windows will overlap, as shown in Due to overlapping, whenever a HOG feature is ready, the SVM module will spend the number of cycles defined by Equation  times thx direction are divided equally among the r in the SVM units matrix is SVM units are proposed to process a subset of detection windows in parallel to deal with this problem. These units are arranged in An example of this distribution of detection windows by SVM units is shown in y of a given SVM unit and the r.Whenever any SVM unit ends all detection windows, it jumps to process a row that is not yet processed by any other SVM unit. The next line to be processed is the sum of the current line The c column. Whenever a feature is available in this FIFO, the column controller sends it to all SVM units of a given column c. The controller also sends the indices of detection windows that belong to this feature. Since the processing follows a sequential and pipeline strategy within the SVM unit, the same feature is sent several times in each clock cycle with the window index value different to cover all the windows that need to be processed. When all indexes are sent, that feature is finished, and the column controller fetches the next available feature in the FIFO.The architecture that processes this processing logic is shown in The detection window processing through the SVM unit starts by accessing the SVM weight index from the detection window index. The SVM weight value, obtained from the weight index, is multiplied by the input feature. This value is added to the temporary confidence value stored in an indexable memory and stored back in this memory. This memory stores the temporary confidence values of all detection windows.b from Equation  step removes several neighboring detections from the same pedestrian.The Tracking step identifies and labels the footpoints that belong to the same pedestrian over several consecutive frames. This step uses an extended Kalman filter (EKF) with vehicle motion compensation for tracking to model pedestrian motion. In addition, the tracking also includes the Hungarian method for the global association and the gate method to exclude unlikely associations.The Geometric Filtering step removes detections that did not meet the pedestrian locality and shape restrictions. Next, the Trajectory Prediction step estimates the future trajectories of the pedestrians and vehicles in several discrete steps. Finally, the Collision Analysis step identifies possible collision positions when pedestrians’ positions are at the same time in the future touching the vehicle. The techniques and parameters used in each of these steps are detailed in .Using OPAE, the SWWrite function writes each new pair of stereo frames in the unidimensional shared array putting each pair of stereo pixels together. On the hardware side, the HWRead module requests these pixels through the CCI protocol (CCI-P)  in burstEach data packet that arrives from the CCI-P is 512-bits wide and can hold more than a pair of pixels. The packages are sorted and stored in a FIFO so that the Unpacker module reads a pair of pixels at a time (RGB pixels of the left and right image). The left image pixel is sent to the proposed PD module, and the pair is sent to the Matching Stereo (SGM) module. Both modules run in parallel. As the results are provided from these modules, they are packed into 512-bit data by the SGM Packer and PD Packer modules.As the memory write is performed one packet at a time, it is necessary that the writing of the results of one module ends so that the results of the next module are written. The HWWrite module does this management and sending of packets to memory. First, it sends the disparity map data. When it finishes sending, it asks the PD module to send the detection results stored in the Serializer module as described in To send or receive several frames, both the software and the hardware implement a synchronization mechanism based on exchanging parameters and control variables. Every time the software finishes writing all the data in memory, it modifies a control variable. When the HWread module recognizes the data available, it requests the reading of these data. Likewise, when the SGM and PD module finishes processing and writing all results in memory, the HWWrite module modifies the respective control variables so that the CPU knows that all results are ready and available for further processing. More communication details can be found in our work .The proposed PD system hardware architecture has been implemented in SystemVerilog language and validated through timing simulations using the ModelSim 10.5b and Quartus 17.1 Standard for synthesis tools. The proposed HW-based PD system (HW PD system) is evaluated in the case study: the PCP system. The database, evaluation strategy, and system parameters are presented to evaluate the pedestrian location. Then, results and analyses are conducted.The database  was adopThe location evaluation considers the lateral (X) and longitudinal (Z) position of the pedestrian obtained from the Geometric Filtering step of the PCP system shown in The test base is divided concerning the distance between the pedestrian and the vehicle. Group 1 is formed by the frames with distances between 7 and 25 m, while Group 2 contains frames between 7 and 50 m. There exist 2666 and 4305 frames for Groups 1 and 2, respectively.To demonstrate improvements when detecting both near and distant pedestrians, first the results from our previous work were reproduced . Two detTwo SVM-Ps were defined to demonstrate similar results as in high-level. The modules To train each SVM-P and the high-level detectors for a given database, positive and negative patches are created from training samples following strategies similar to ,34. The The following parameters are common to all evaluated systems. The dimensions of the HOG cell and block in pixels are, respectively, 8 × 8 and 16 × 16. The detection window stride in both directions is 8 pixels. The number of bins of the HOG histogram is set to 8. The pyramid scale parameter The parameters of the SGM module  are the The detectors are analyzed by miss rate versus false positive per image (FPPI)  as showny-axis) and FPPI (x-axis) are detector results. The detector parameter to obtain these two results is the detector confidence It is essential to clarify that the miss rate , estimated through an average of 1000 frames processed with outliers removed due to operating system scheduling.A significant result noted in The effectiveness of the PCP system on the collision prediction was evaluated with the improvement of the PD system processing performance in hardware. This evaluation defines the database used, evaluation strategy, and the configuration of the systems being assessed, and presents results and analyses.Since it was not possible to find a collision database, an evaluation database was built from a kind of scenario defined on , as showThe parameters for the scenarios created are the vehicle velocity ash risk .Following , the valAll the combinations between The PCP system’s efficiency is assessed by checking the distance at which the system predicted a collision for the first time from the moment the pedestrian appeared (the first collision prediction). The safe distance supports a prediction assessment that ensures that the vehicle will not collide with the pedestrian if the system predicts the collision above that distance. This safe distance  is defin−4.5 m/s2. These vThe HW PD system (W64_H128_W48_96_D3_HW) was evaluated within the PCP system . In this same PCP system, the HL PD system (W64_H128_W48_96_D3_HL) was also assessed . The YOLOv3-based PD system was also evaluated (PCP/YOLOv3). HL-PD-based and YOLOv3- based PD systems run on RTX 2070 GPU with 8GB of memory. The processing rates of the PCP/HW-PD, PCP/HL-PD, and PCP/YOLOv3 systems in the synthetic database are 60 FPS, 30 FPS, and 12 FPS, respectively. Since the database sampling rate is 120 FPS, capture intervals were defined that correspond to the actual rates of each PCP system. The intervals for the PCP/HW-PD, PCP/HL-PD, and PCP/YOLOv3 are 2, 4, and 10 frames, respectively.The PCP system parameters are the same as defined in fined in  with tesThe training strategy is the same as described in It can be noted in Although it is not always possible to guarantee safe distance, it is possible to note an increase in decision-making distance with the HW PD system (PCP/HW-PD) at all speeds and TTC configurations. To better evidence this distance gain, the average, at each velocity, of the distance differences between the proposed system and two others  for all TTC values were calculated, as shown in It is possible to observe in This difference increases almost linearly with the vehicle speed. This is due to the response time of the proposed detection system, which is shorter than other systems. Since several frames are needed for collision estimation, this time difference leads to an increased gain in the distance with faster vehicles. This linear growth brings more safety to the pedestrian, especially in cars at higher speeds. Efforts to reduce the response time of the PCP system will increase the safety distance even further. Specifically, in the PCP system proposed in this work, solving the communication bottlenecks due to the platform, as mentioned in As may be observed in The result in terms of MWPS of the proposed PD system was measured in the prototyping platform, considering the platform communication overhead. Our system has a higher MWPS than most of the works. Although  achievesThis work proposed an FPGA-based implementation of the pedestrian detection (PD) system based on HOG and SVM techniques, supporting image pyramid and detection windows of different dimensions. It was demonstrated that with detection windows of various sizes, the PD system’s miss rate was reduced by at least 6% compared to another system processing detection windows of unique dimensions. Specific strategies for each step are proposed to ensure the throughput of one pixel per clock cycle. For the most critical SVM module, a novel approach was proposed to process many detection windows through units using parallel and pipeline processing to process a different set of windows. This approach provided a gain in processing performance compared with other works and a constant performance independent of the pyramid depth and quantity of different-size detection windows. The processing performance gain of the proposed PD system inside the PCP system managed to avoid more collisions compared to lower systems and ensured an improvement in decision-making distance of a maximum of 3.3 m. Future works include scalability improvements to the proposed PD system to process more windows of different dimensions and more pyramid levels.


Nowadays, consumer electronics offer computer-vision-based (CV) measurements of dynamic displacements with some trade-offs between sampling frequency, resolution and low cost of the device. This study considers a consumer-grade smartphone camera based on complementary metal-oxide semiconductor (CMOS) technology and investigates the influence of its hardware limitations on the estimation of dynamic displacements, modal parameters and stiffness parameters of bolted connections in a laboratory structure. An algorithm that maximizes the zero-normalized cross-correlation function is employed to extract the dynamic displacements. The modal parameters are identified with the stochastic subspace identification method. The stiffness parameters are identified using a model-updating technique based on modal sensitivities. The results are compared with the corresponding data obtained with accelerometers and a laser distance sensor. The CV measurement allows lower-order vibration modes to be identified with a systematic (bias) error that is nearly proportional to the vibration frequency: from 2% for the first mode (9.4 Hz) to 10% for the third mode (71.4 Hz). However, the measurement errors introduced by the smartphone camera have a significantly lower influence on the values of the identified stiffness parameters than the numbers of modes and parameters taken into account. This is due to the bias–variance trade-off. The results show that consumer-grade electronics can be used as a low-cost and easy-to-use measurement tool if lower-order modes are required. Information about structural stiffness, which is crucial in structural health monitoring (SHM), is better reflected by dynamic displacements than by accelerations, as the latter are strongly affected by other parameters, e.g., mass . NowadayTo be installed, a camera does not require physical access to the structure. Hardly accessible structures can be observed remotely using zoom lenses ,6. In coCompared to GPS technology, which also does not require a physical reference point near the monitored structure, CV measurements are much more accurate .As opposed to traditional point-wise sensors, a camera is able to simultaneously capture the motion of multiple points . MoreoveFeng and Feng indicated several advantages of CV displacement measurement over techniques employing traditional sensors :To be inApart from the advantages of CV measurements, some weaknesses of this technique should also be noted. Measured higher-frequency components of motion often become highly contaminated by noise due to equipment limitations such as insufficient camera resolution or a relatively low sampling frequency . Thus, sTemplate matching methods are the most popular displacement estimation techniques due to the possibility of achieving subpixel-level accuracy. Generally, in template matching, the tracked object is represented by a preselected image region, namely, a template, of a selected frame of the video . The displacement of this template is determined in the subsequent video frames by searching for and matching this template with the most similar region within a search area in the current video frame, called the region of interest (ROI). The template matching process is illustrated in Based on the matching technique, template matching methods can be classified into area-based and feature-based template matching . In areaErrors caused by the limited resolution of the camera can be reduced by using subpixel techniques . Feng etCamera calibration is required to transform the displacements in the video (expressed in pixels) into physically meaningful information. In the simplest case, if the direction of the camera view is perpendicular to the filmed surface, then only a scaling factor is required. This can be determined from the known physical dimensions of the filmed structure and its pixel size in the video frames . When thAnother important issue in CV measurement is the camera motion caused by ground vibration. This problem is not always significant, as shown by Feng and Feng, where the error resulting from camera motion was negligible . HoweverThere are also other sources of errors. In remote measurements from a significant distance, heat haze can distort the refractive index of the air and affect the measurement accuracy . The widDue to the mentioned advantages of low-cost cameras, especially smartphone cameras, they are frequently used for estimation of displacements and identification of structural dynamic properties ,32,33,34The dynamic displacement estimation techniques discussed above are often used for identification of modal parameters of structures, which can subsequently be used for model updating. Among the variety of methods for experimental modal analysis, stochastic subspace identification using time-domain data (SSI-DATA) is widely recognised as accurate and robust . It is dExperimentally identified modal parameters are often used in the model updating procedure. Model updating generally has one of two aims: (1) calibration of the model, which aims to provide accurate reproduction of the output of the real system, or (2) identification of unknown structural parameters .It is known that the results of a model updating procedure are affected not only by the measurement noise and modelling errors but also by ill-conditioning of the problem, and thus numerical regularisation of the solution  enhances the quality of the solution. In the widely used method based on modal sensitivity, which is also called mode-matching, regularisation is ensured by an additional term in the objective function that involves prior knowledge about the structural parameters being identified ,38. CondThe literature review shows that significant effort has been devoted to enhancing the performance of CV measurements. A similar trend can be observed in SHM and CV-SHM. Similarly, an increase in the number of studies that employ low-cost cameras in smartphones can be observed due to the growing potential of cost-effective measurements. However, to the knowledge of the authors, there are still only a limited number of works that discuss the influence of limitations and measurement errors of computer vision equipment on the accuracy of structural identification procedures. In particular, low-cost consumer cameras and their use in ill-conditioned identification problems with a limited number of measurement data are not fully investigated.The present study investigates the influence of CV measurement uncertainties and equipment (a CMOS smartphone camera) limitations on the accuracy of: dynamic displacement estimation, identification of modal parameters and the model updating procedure. The paper is structured as follows. In This subsection describes the investigated laboratory-scale frame structure shown in When all joints are in the locked state, the investigated structure does not exhibit any considerable nonlinear behaviour if the amplitude of motion is relatively low. Thus, a linear FE model and a linear system identification method are applied in this study. More details are shown in the next section.A linear FE model ures see b, and th3, the mass of the joint is 2.Both FE models have 139 DOFs. In-plane beam elements with 6 DOFs based on Euler–Bernoulli beam theory are employed. Cubic shape functions are used. The joints are represented as rigid bodies by using appropriate offsets between the semi-rigid nodes. The mass matrix In this section, all steps of the investigated CV methodology for identification of structural parameters are discussed. The order of the steps is presented in In the present study, a Samsung Galaxy S20 FE smartphone equipped with a high-speed camera based on CMOS technology is used to register structural behaviour. The technical data of the camera are listed in Aiming at achieving a trade-off between the largest field of view possible as required for multi-point measurements and satisfactory resolution, the smartphone is set to capture artificial targets located between the middle and the tip end of the structure. The distance from the smartphone to the structure is not measured. The recorded part of the structure with the manually selected templates around each measurement location (“node”) and the automatically generated ROIs is shown in Two depths of the image plane can be distinguished: one for the artificial targets located on the beams and another one for the targets on the joints compare . Thus, tIn the present study, CV measurements are performed in laboratory conditions with good illumination during the experiments. Changes in the illumination and the scale can be considered negligible in the provided measurement conditions; hence, the robustness of the feature-based template matching with respect to these sources of error is not necessary. Thus, due to its high accuracy, the area-based template matching method is employed. The template is matched with the ROI by maximization of the ZNCC :(3)ZNCC−f^mth identified system eigenvalue, and The stabilization diagrams obtained with a smartphone camera can be highly contaminated, and thus, clustering of the stable poles according to system of inequalities  may resuEquation , but theMACϕ^(mi)mth identified eigenvalue, Stiffness parameters are found with the modal-sensitivity-based method by solving the optimization problem  follows:F(θ)=eMTVθ=SIt is worth noticing that the variances In this section, the results of the methodology described in previous sections based on CV measurement with the aid of a smartphone camera are elaborated. First, the benchmark data based on accelerometers are described. Next, the accuracy of CV measurement with the aid of a smartphone camera is demonstrated. Subsequently, the identified modal data obtained with the smartphone camera and accelerometers are compared. Finally, model updating based on these two data sets is performed, and the results are compared.During the tests, Nodes 13, 16 and 19 were rejected due to the fact that the retraction of the rubber hammer after the hit causes contamination of the measurement by the hammer’s shadow. This phenomenon is demonstrated in In this section, accelerometer-based identified modal parameters are marked with the subscript “acc”, e.g., Accelerometer-based modal parameters were identified using the SSI method with the aid of an LMS-SCADAS system and the LMS Test.Lab software. All 26 locations of bidirectional accelerometers were selected to provide the results for comparison with the CV measurements. Accelerometers B&K 4507 B 004 were used in this study. Five experimental sessions were conducted, resulting in five data sets. The natural frequencies of these data sets are shown in The modal parameters used later for model updating are calculated by averaging over all data sets; see mth mode shape, and it is calculated as follows:ith element of the mth mode shape, and ith measured DOF of the mth mode shape The normalised standard deviation mth mode being equal to The physical sense of Equation  with SmϕIn this subsection, the accuracy of CV measurement of dynamic displacement is demonstrated. The CV measurement of transversal displacement of Node 2  is compaThe vector containing the displacement time series lem (P2) :(P2)find follows:J(θ)=12WDThe result is visualised in ecv=std. Such aIt is evident that the results are dependent on the number of the available measured modes. Both for the CV- and accelerometer-based data, the unknown parameters have different values in each measurement data case. Simultaneously, the standard deviations of the unknown parameters are smaller with the increase of the available measurement data. These observations are visible for both considered parameterizations (two and a single unknown parameter). However, the single unknown parameter seems to be less sensitive to the amount of the measurement data, and it has a lower standard deviation than the corresponding results obtained for the two parameters arameter b.The parameter  than θ1 a,e due t than θ1 b. HoweveThe considerations above refer both to the CV- and accelerometer-based measurement data. Moreover, the number of available identified modes has a greater influence on the values of the unknown parameters than the errors in the CV identification of the modal parameters described in A comparison of the error metrics between the numerical and accelerometer-based identified modal parameters with the corresponding error metrics obtained for the CV-identified parameters for various available identified modes when the FE model is updated with the two unknown parameters It is evident that the errors between the numerical and identified modal parameters for the simplified parameterization of the FE model  are onlyIn this subsection, the obtained results are discussed in the context of their applicability both for real-world structures and laboratory experiments.The third CV-identified mode has a natural frequency above 1/2 of the Nyquist frequency due to the limited sampling frequency of the smartphone camera. This results in considerable systematic error. The third mode shape also becomes noisy. However, real-world structures are often characterised by natural frequencies of much lower values; hence, they are less subject to estimation error when a camera with a limited sampling frequency is used. In , a camerRegarding the parametric identification of the structural properties, in the investigated case, the number of the available modal parameters used in the optimization procedure affects the results more than the uncertainties resulting from CV measurement. A similar observation is given by Blachowski, who indicated that an increase in the number of measured modes is more profitable than an increase in the number of measurement locations . This isIn recent times, machine and deep learning approaches in SHM have become more and more popular. Artificial neural networks can enhance the performance of SHM techniques, especially when the monitored structure is large-scale and exhibits a nonlinear relation between the damage and measured output . The metThe employed consumer-grade smartphone camera with the “slow-motion” mode selected, full HD resolution and a sampling frequency of 240 Hz allowed us to estimate structural displacement with an error two times higher than that when using LDS (10.82% vs. 5.14%), where a fusion of LDS- and accelerometer-based data were taken as the benchmark.The selected mode of recording allowed the first three vibration modes to be identified. The MAC between the CV- and the corresponding accelerometer-based identified mode shapes had satisfactory values: 0.99, 0.98 and 0.92. However, the identified natural frequency was underestimated: the first natural frequency (9.39 Hz) was underestimated with an error of 2.03%, whereas the third one (71.40 Hz) had an error of 10.43%. This error was recognised as systematic, since the variances of the CV-identified modal parameters were not higher than the accelerometer-based ones.The unknown parameters representing the stiffness of the bolted connections were identified with the modal-sensitivity-based model updating technique. The parameters identified using CV measurements had slightly lower values than the ones based on accelerometers, which was due to the underestimated natural frequencies. However, it was shown that the number of identified modes taken into account had a significantly greater influence on the unknown parameters than the systematic error of the CV-identified natural frequency. Reduction of the number of unknown parameters reduced their sensitivity to the number of available measurement data due to a more profitable bias–covariance trade-off.Despite the noticeable errors obtained with the use of the smartphone camera, it should be noted that the cost of consumer-grade equipment is significantly lower than that of accelerometers and a data-acquisition system with integrated software. Moreover, a camera allows performance of “by hand” and multiple-point measurements at once.This study investigated the influence of the equipment limitations and errors of CV measurement on the identification of both modal parameters and unknown structural parameters that represented stiffness of structural bolted connections. The conclusions can be drawn as follows:The results provide insight into the expected error types and levels, not only for the directly measured quantities but also for the parameters identified in structural dynamics, when using low-cost consumer-grade electronics.