There are no brachytherapy treatment planning systems (TPS) exclusively for the treatment of cervical tumours, so general-purpose TPSs are used. However, these treatments have some particular features concerning the treatment of other pathologies, especially in the case of exclusive use of MRI as an imaging modality and the presence of gynaecological applicators in combination with an interstitial part. That is why it is essential to review the latest versions of commercial TPSs to find the potential features to improve with the help of a group of experimented medical physicists and radiation oncologists. Furthermore, after reviewing the recent literature for advances applicable to cervical brachytherapy and through his own clinical experience, possible improvements are proposed to software providers for the development of new tools.Intracavitary brachytherapy , plays an essential role in the curative intent of locally advanced cervical cancer, for which the conventional approach involves external beam radiotherapy with concurrent chemotherapy followed by BT. This work aims to review the different methodologies used by commercially available treatment planning systems (TPSs) in exclusive magnetic resonance imaging-based (MRI) cervix BT with interstitial component treatments. Practical aspects and improvements to be implemented into the TPSs are discussed. This review is based on the clinical expertise of a group of radiation oncologists and medical physicists and on interactive demos provided by the software manufacturers. The TPS versions considered include all the new tools currently in development for future commercial releases. The specialists from the supplier companies were asked to propose solutions to some of the challenges often encountered in a clinical environment through a questionnaire. The results include not only such answers but also comments by the authors that, in their opinion, could help solve the challenges covered in these questions. This study summarizes the possibilities offered nowadays by commercial TPSs, highlighting the absence of some useful tools that would notably improve the planning of MR-based interstitial component cervix brachytherapy. In the last few decades, brachytherapy  has been one of the cornerstones for the treatment of gynaecological lesions, being one of the most widely used, and successful, therapeutic techniques to be considered in these cases. Within gynaecological BT, cervix treatments have experienced the greatest development in recent years. Numerous authors have reported that intracavitary BT plays an essential role in the curative intent of locally advanced cervical cancer management, for which the conventional approach involves external beam radiotherapy (EBRT) with concurrent chemotherapy, followed by BT. Excellent local control rates are achieved with this scheme, as it has been demonstrated by different studies on the curative intent of locally advanced cervical cancer ,2,3.In the past decades, the development of new EBRT techniques, such as Intensity Modulated Radiotherapy (IMRT), Volumetric Modulated Arc Therapy (VMAT), Stereotactic Body Radiotherapy (SBRT), or Proton therapy (PT), have broadened the choices of a treatment scheme. In fact, the data of BT use in cervical cancer in the United States (USA) published by Han et al.  revealedThe gynaecological working group of the GEC ESTRO (GYN GEC-ESTRO), formed in the year 2000, is committed to move from 2D BT based on A points to 3D BT based on images, incorporating magnetic resonance imaging (MRI) in the BT planning of the cervix. A new concept of volume definition is proposed, taking into account the topography of the primary tumour at diagnosis and after response to radio-chemotherapy, as well as dosimetric aspects. GYN GEC-ESTRO published its first two recommendations in 2005 and 2006 ,9, its nOnce adopting the use of MRI, it has been evidenced that the interstitial component added to the intracavitary one achieves a higher dose conformity and organs at risk sparing. It also allows the use of different combined applicators options depending on the manufacturer ,23. PresRadical locally advanced cervical cancer treatment comprises pelvic EBRT with concomitant chemotherapy, following BT. The EBRT dose is usually 45–50 Gy in 1.8–2 Gy fractions. Lymph nodes, parametria, or pelvic sidewall boosts up to 55–64 Gy are used, depending on the clinical case and the experience of each individual centre ,19. AlthAs recommended by ICRU 89, the difference in the biological effects of the EBRT and BT fractions makes it necessary to define a parameter to evaluate them globally. The previously referenced publications make use of the dose equivalent of 2 Gy per fraction (EQD2), using the linear-quadratic model with α/β = 10 Gy for tumour effects and α/β = 3 Gy for late normal tissue damage, assuming a half-time repair time of 1.5 h. Therefore, following the ICRU 89 recommendations, EQD2 is the chosen option to sum the absorbed dose to report dose for planning aims, prescriptions, and delivered dose.Clinical departments use external tools, such as the GEC-ESTRO spreadsheet, to calculate the resulted EQD2 for each patient. None of the current brachytherapy treatment planning systems (TPS) consider the fully CTV-OAR dose balance during the optimization, becoming a user-dependent procedure mostly based on the physicist experience.Ideally, the reconstruction of the applicator geometry and its matching with the patient anatomy should be done in the same image study in which the radiation oncologist defines the CTV and OARs volumes in T2 weighted MRI. However, there is an important limitation to that technique when using dummies to identify the potential source trajectory for the interstitial component using plastic catheters. Namely, the void signal produced by the air-filled interior of the applicators makes it difficult to set the catheter tips. Therefore, the use of other image modalities is sometimes beneficial in the reconstruction process.Therefore, the aim of this work is to review the different methodologies used by all commercially available TPSs to solve the main planning issues in an exclusive-MRI-based cervix BT with interstitial component treatment. Moreover, some desirable or convenient practical aspects to be implemented in future TPS versions are outlined from both radiation oncologist and medical physicist perspectives.The authors of this review are radiation oncologists and medical physicists with extensive experience in MRI-based cervix HDR brachytherapy treatments with an interstitial component, representing the Spanish centres with the largest number of implants performed. The three TPSs analysed were Brachyvision v16.0 from Varian , Oncentra v4.6.2 from Elekta , and Sagiplan v2.2 from BEBIG . Most of the authors are users of Oncentra Brachy, with just one user of Brachyvision; there are no Sagiplan users.The reviews of the TPSs were done based on the authors’ clinical experience and interactive demos provided by the software manufacturers. In these demos, the TPS version tested was the latest one commercially available, including also all the new tools currently developed to be included in future releases. After an introduction to most of the utilities and tools of the planning systems, the specialist from the vendor company  was asked to answer some of the issues that are routinely faced in a clinical environment and that will be briefly described below. In order to follow a systematic approach on the evaluation of each TPS capabilities, demos were carried based on the questionnaire summarized in The work plan has been complemented by a review of the literature on this specific topic. The bibliography review methodology was based on a keyword search in the PubMed database of publications from the last ten years. Such keywords include MRI, cervical cancer, HDR brachytherapy, catheter reconstruction, dose accumulation, etc. On the other hand, more specific keywords, such as deep learning, electromagnetic tracking, auto-segmentation, and synthetic-CT, were also included. The study aims to highlight the general shortcomings of the TPSs and the possible improvements that could be made to them in the opinion of a group of experienced users. However, it is not the authors’ intention to make comparisons between them, much less to recommend a ranking.Aside from the verification of the applicator’s geometry, the medical physicist must check the TPS calculation of the dose , the applicator’s geometry within the digital libraries, and all the tools involved in a treatment planning. Digital libraries reproduce the geometry of the applicators. However, the real path of the source could differ from the symmetric axis of the applicator, which is usually the path introduced by the manufacturer in the corresponding libraries . This efTPSs should be compatible with these generic models and facilitate the MBDCA commissioning by implementing the tools necessary to perform an in-depth comparison of the 3D dose distributions. Finally, TPSs must easily perform the quality control tests suggested in the guidelines ,30,31.GYN IGABT image registration would be a fundamental tool to use in different phases of the planning process, such as in applicator reconstruction, volumes definition or propagation, dose accumulation in multifraction BT, or dose accumulation of EBRT + BT.In the case of applicator reconstruction or when transferring the target volume contours of a first MRI implant to the scans used for a second implant, for images with the applicator in place, a rigid registration of the applicator is recommended ,32. RigiIf the brachytherapy treatment consists of more than one implant, it would be very useful to include the dose distributions of previous implants in the optimization.Since the brachytherapy applicator completely deforms the anatomy compared to the EBRT images, rigid registration has been questioned as a valuable method. Deformable registration could, in principle, combine the dose from each tissue voxel in the EBRT fractions with the corresponding one for each BT fraction.A significant issue for cervix segmentation is the presence of the applicator. Such an applicator causes deformation of the surrounding tissues, hindering the correct segmentation . AdditioDetermining the source path and the most distal dwell position, together with the matching of the anatomical geometries, are the catheter reconstruction objectives . The CT An alternative to this modality of reconstruction is the use of applicator libraries containing accurate 3D models. The corresponding applicator can be selected and then displaced and rotated until it matches the image using reference points located both in the image and the model. Thereby, the source path and the most distal dwell position are clearly defined. This method is only valid for rigid applicators and, therefore, excludes the interstitial part.As discussed in Q4, the use of applicator libraries is only helpful in the case of rigid applicators and therefore the use of applicator libraries is, in principle, not possible for the interstitial part. Thus, direct reconstruction is the only way to make such reconstruction. In contrast to the case of needles on CT images, identifying needle trajectories and needle tips is still an open issue in MRI.In the case of the DR modality, the determination of the most distal dwell position involves an extra challenge due to the finite slice thickness. Even though a slice thickness lower or equal to 3 mm is suitable for MRI images, the acquisition protocols of a given institution may not offer such a possibility while still needing to achieve millimetric precision on the reconstruction. This precision goal is especially important for the tip position. A possible solution is to add reconstructed images between two slice thicknesses, thus reducing the uncertainty.Objective and tolerance doses on locally advanced cervix cancer are expressed in EQD2 and not in physical dose units. This accounts for the fact that two treatment modalities with different fractionations and biological effectiveness are intended to be combined (EBRT + BT).Independently of the method, optimization is based on an iterative process in which the medical physicist and radiation oncologist verifies whether all objective and tolerance doses (in EQD2) are fulfilled. At the time of publishing this review, these iterations are done using an external spreadsheet.As previously mentioned, the most used parameter for combining the biological effect of EBRT and BT fractions is EQD2. Therefore, mandatory and optimal constraints should be expressed using that quantity. If TPSs can combine the doses of the different fractions in an agile way, an excess or lack of dose can be detected in the EBRT phase and compensated during the BT phase. In addition, for the definition of the intermediate-risk clinical target volume (CTV-IR), it is necessary to transfer the pre-EBRT GTV to the MRI of the BT treatment (ICRU 89) .The most recent protocols (EMBRACE II and ICRU 89) suggest de-escalating vaginal doses  and controlling the contribution from the interstitial component (less than 20–30%) ,25. This90, D2cc, etc.). Moreover, the capability of the optimizers to control the dwell time gradient/homogeneity and the weight of each component  was also assessed.Once the catheters are reconstructed, the dwell times must be chosen to fulfil the prescribed dose to the target volumes. To this end, several optimization methods have been used. This question was included to get details on the different optimization methods available for each TPS and their limitations. Specifically, the specialists were questioned about the inclusion of inverse-optimization algorithms and the possibility to aim the optimization towards the dosimetric metrics suggested to be reported (D100) and maximum doses. Instead, more robust dosimetric metrics were gathered: D98, D90, D2cc, and D0.1cc, among others. The control of the DVH resolution limitations is extremely important in the case of the OAR for which D2cc and D0.1cc are suggested to be reported. This question was aimed to identify the strategies of the different TPSs to be able to identify the doses deposited for volumes up to 0.1 cm.Following the advice of ICRU 89, the EMBRACE group decided to abandon the dose report based on minimum .A Dorbidity ,38. MazeIntroduction of MBDCAs capable of accounting for tissue and applicator heterogeneities in brachytherapy has been a major development in brachytherapy treatment planning in recent years. Their emergence and the protocols for early adopters was addressed by AAPM, ESTRO, ABS, and ABG Task-Group 186 (TG186) . TG186 eAfter the specialists presented the demonstrations, they answered the questions raised by the authors. Their answers to the questionnaire are summarized in this section following the scheme outlined in the Materials and Methods section, together with the authors’ comments.Only one TPS allows the modification of the source path within the applicator’s library. It is essential for curved applicators, such as rings or ovoid ones, where the source usually moves close to the applicator wall and far from the axis of symmetry.Out of the three TPSs analysed, only two of them include the possibility of performing MBDCA calculations. None of the test cases available at the time of writing this manuscript are clinical cases, with WG-DCAB currently developing specific test cases for GYN. However, Test Case 4, although it does not fully resemble a clinical situation, incorporates a shielded generic GYN application, and therefore it is the most interesting one for the GYN clinical user to commission his/her TPS. None of the TPSs analysed incorporate a complete set of tools to perform that commissioning process, requiring the clinical user to use external tools, such as BrachyGuide  or AMIGORegarding the commissioning of the dose calculation by TG43, the authors believe that the verification procedure is not as operational as it should be. The dose that the TPS returns for specific points will be compared with the consensus along–away tables using the TG43 protocol functions for a given source model. However, these points can only be entered via the keyboard and one at a time. A more agile and less error-prone method is desirable.All three TPSs allow rigid registration, although only one includes the possibility of deformable image registration.In theory, deformable registration would match and sum each voxel dose combining each fraction of EBRT with the corresponding contribution of BT. However, none of the available TPSs include the technology required to track the effect on biologically meaningful structures within target volumes and OARs without using external compatible software. For this reason, GEC-ESTRO recommends assuming that the organ walls adjacent to the brachytherapy applicator and target volumes received the EBRT prescribed dose. Such a conservative approach, already widely used in clinical practice, has provided reliable results for both target and non-mobile OARs, despite being a rather important assumption. In addition, following one of the EMBRACE II recommendations for the planning of EBRT, a control region around the high-risk clinical target volume (CTV-HR) of 10 mm thickness could be generated to establish a dose homogeneity requirement in order to avoid hotspots in the OARs close to the CTV-HR, which are likely also to receive a considerable BT dose. Special attention should be paid in EBRT cases with multiple PTVs and doses due to the largest OAR D2cc dose.None of the TPSs allow removing the ring or ovoids from the MRI. In the case of using applicator libraries for reconstruction, information on the position of the applicator together with its composition could help resolve this issue in future versions of the TPSs.All TPSs analysed have access to intracavitary applicator virtual libraries for catheter reconstruction. As previously discussed, it is useful to facilitate and reduce the uncertainty in the reconstruction phase. The DR is an alternative to this type of reconstruction and the only solution for non-rigid applicators, especially for the commercially available interstitial part. However, taking the example of reconstruction of a ring applicator that includes a tandem, both parts should be treated as two different applicator library elements. Nevertheless, the tandem is joined to the ring, so they are not strictly independent and may offer additional geometrical constraints that might be used to improve its reconstruction. It would be desirable to further develop this idea.All specialists mention the intention of improving the auto-placement algorithm on MRI image sets because they recognized the inherent uncertainty in all cases. TPSs already incorporate such tools, but they are only accurate when used on CT images. Hrinivich et al.  have devIn the case of the reconstruction of the interstitial part, TPSs allow reconstructing the needles through the black area left by the plastic needles and the offset between the tip and first dwell position. However, in some cases the black area is not seen clearly enough to perform such reconstruction efficiently.Along the same lines, as mentioned for the intracavitary part, the needle path is not entirely independent of the rest of the applicator. Richart et al.  suggest Taking the free length concept, Otal et al.  developeIn parallel to solutions based on applicator libraries, some groups continue to work on direct reconstruction methods. Although needle diameter is a challenge for the development of dummies, there are possible solutions already under development. Shaaer et al.  have tesTwo out of the three TPSs studied include the probability to access “interpolated slices” between the acquired ones. The third TPS has needle models in its applicator library but just in the case of straight ones. All specialists mentioned the manufacturer’s intention of enhancing the auto-placement algorithm on MRI image sets because they recognized the uncertainty in current TPS versions.Only one of the TPSs has the option to import the dosimetric information of the fractions delivered. In this case, the DVH information is directly added to the BT planning and transformed to EQD2 units based on α/β values introduced by the user. There is no summation of dose distributions and the DVHs are only extracted from the EBRT fractions. It would be more convenient if such an iterative process were integrated into the TPS and incorporated into the optimization phase.To obtain the biological effect of the EBRT and BT phases in terms of EQD2, it is necessary to register all BT fraction delivered as accurately as possible . AlthougSwamidas et al.  and Kim Current TPSs have limitations when calculating cumulative dose distributions and when deriving a composite DVH from EBRT and BT plans in EQD2. It would be desirable to promote the research and evolution of deformable registration algorithms and adapt them to the complexities of cervical radiotherapy treatment planning.All TPSs includes several tools for modifying the dwell times: manual variation of times, normalization to points or reference lines, general renormalization, and graphical optimization, among others. The ability to lock individual catheters and dwell positions, making the dwell times non-modifiable, was only included in one TPS. This tool is helpful since it facilitates the control of the weight of each component .90, D2cc, ...) is offered in all TPSs. The D2cc value extracted from the DVH is also reported. Even though these optimization modules have the potential to offer clinically acceptable dose distributions, none of the TPSs provided the required control of the dose gradient around needles despite having specific parameters to modulate it. Another interesting tool already incorporated in the EBRT inverse optimization algorithms is the possibility of starting from a solution given by the user and using it as an initial state in the optimization process. BT TPSs will benefit from incorporating a similar feature.Inverse optimization based on the objective dosimetry metrics  to dispense with external tools.Due to the relationship of “hot spots” in OARs with their toxicity, a suitable tool for locating such spots would be desirable.Finally, integrating deformable registration algorithms in the TPS would be very helpful.In summary, the main points that commercial TPSs would be requested to improve significantly in their planning procedures would be:The main objective of this study is to propose improvements and point out the gaps in the existing TPSs. External software compatible with the TPSs that could help improve the issues outlined here were excluded from the study, including those scripting environments within the TPSs that may allow the user to program personalized tools. As discussed above, this review is based on the clinical expertise of a group of radiation oncologists and medical physicists. Interactive demos provided by the software manufacturers and delivered by specialists from the supplier companies were also used. These specialists were asked to propose solutions to some of the challenges through a questionnaire. Possible limitations of this methodology are the following. It is expert- and specialist-opinion based, therefore personal biases are always a risk. In this case, considering that it is a relatively large group of professionals working in different institutions , such an issue is diminished. Another possible limitation is the fact that not all experts have used all TPSs in their clinical practice, something unavoidable considering the realities of clinical practice. The use of questionnaires also might limit the study, since they inherently have selection and sampling biases that might preclude the discovery of unforeseen situations.In addition, this modest multicentre and multidisciplinary study is fundamentally aimed at pointing out deficiencies and limitations rather than quantifying their consequences.There is a segmentation software that incorporates more advanced tools for image registration that include utilities for dose summation between different fractions. It would be desirable that such, or similar, tools eventually make their way to the TPSs.Other tools, this time related to the segmentation of clinical volumes and organs at risk, that would be interesting to incorporate into brachytherapy-specific TPSs would automate such segmentation by means of convolutional neural networks. These are already being incorporated into the clinical workflow in radiotherapy, and there are recent works in the literature explicitly oriented to the cervix case ,55,56, aAn exciting topic in external beam therapy is the generation of synthetic CTs from MRI. The main objective of the generation of such CTs is to segment tumours and organs at risk in the MRI and obtain the electron densities necessary for dose calculation . These sIn the last years, several research groups have been working on electromagnetic tracking systems ,59,60,61Some studies have recently been published investigating the feasibility of deep learning-based algorithms for semi-automated reconstruction of interstitial catheters during MRI-based gynecological HDR .This review has been performed by radiation oncologists and medical physicists with extensive clinical experience in MRI-based cervix HDR brachytherapy treatments with an interstitial component, representing the Spanish centres performing the largest number of implants. The main goal being reviewing the different methodologies used by all commercially available TPSs to solve the main planning issues for such a technique. Some desirable or convenient practical aspects to be implemented in commercial TPSs are outlined from the perspective of both radiation oncologists and medical physicists. This study also highlights the absence of some useful tools that would notably improve the planning of MR-based cervix brachytherapy with an interstitial component.


In this contribution, we present a simple and intuitive approach for estimating the exterior  calibration of a Lidar instrument with respect to a camera as well as their synchronization shifting  during data acquisition. For the geometrical calibration, the 3D rigid transformation of the camera system was estimated with respect to the Lidar frame on the basis of the establishment of 2D to 3D point correspondences. The 2D points were automatically extracted on images by exploiting an AprilTag fiducial marker, while the detection of the corresponding Lidar points was carried out by estimating the center of a custom-made retroreflective target. Both AprilTag and Lidar reflective targets were attached to a planar board  following an easy-to-implement set-up, which yielded high accuracy in the determination of the center of the calibration target. After the geometrical calibration procedure, the temporal calibration was carried out by matching the position of the AprilTag to the corresponding Lidar target (after being projected onto the image frame), during the recording of a steadily moving calibration target. Our calibration framework was given as an open-source software implemented in the ROS platform. We have applied our method to the calibration of a four-camera mobile mapping system (MMS) with respect to an integrated Velodyne Lidar sensor and evaluated it against a state-of-the-art chessboard-based method. Although our method was a single-camera-to-Lidar calibration approach, the consecutive calibration of all four cameras with respect to the Lidar sensor yielded highly accurate results, which were exploited in a multi-camera texturing scheme of city point clouds. In recent years, there has been an increasing demand for efficient data utilization derived from various sources. The fusion of 3D range data acquired from laser scanners with color information from RGB cameras is a typical example of multimodal data exploitation, commonly adopted in 3D city modeling scenarios. Several mobile mapping systems (MMSs) have been reported and used ,2,3, whiAn indispensable procedure during multimodal data elaboration is the accurate recovery of the sensor’s relative orientation (rotation and translation), a process also known as external calibration ,5,6,7. DAmong methods that utilize special targets for Lidar–camera calibration ,10,11,12Having fixed the interior orientation of the camera, the method of Zhang et al.  was the Padley et al.  exploiteIn a similar manner, Zou et al.  recentlyOn the other hand, in the past years, there has been a growing interest in targetless calibration employing scene feature correspondences , mutual (i)The achievable accuracy of object-based calibration techniques is more controllable by the user compared with that of targetless approaches. An extra capture of the target at a new viewing position may significantly increase the calibration result with practically zero overhead of capturing and processing time.(ii)The construction of the target should be simple for non-specialized users and should include readily available and low-cost materials. Planar boards are prevalent calibration targets; however, complicated patterns, such as those with circular holes ,17, requ(iii)The dimensions of the calibration object are arbitrary (unstructured object) in contrast to most target-based approaches .Along with geometrical calibration, the user can perform temporal calibration for the synchronization of the camera and the Lidar frames. This is a common issue in systems that lack the capability for hardware synchronization, and it can be performed as a second step after geometrical calibration. Finally, our calibration algorithm is provided as an open-source GUI software implemented in the ROS platform, making it easily accessible to users for research, development, and production purposes. The code is available at Feature Extraction phase, 3D to 2D point correspondences are defined among the Lidar and camera reference systems. Using the extracted point pairs, the algorithm estimates the extrinsic parameters of the camera in the Lidar coordinate system.The proposed approach is based on the establishment of dense and highly accurate pairs of corresponding points among Lidar scans and RGB images, aiming at the recovery of the exterior orientation parameters  of the camera with respect to the Lidar sensor. The internal parameters of the camera were acquired first by exploiting open-source camera calibration toolboxes available in the computer vision community .A key component of this endeavor is the exploitation of a calibration object that is easy to set up and ensures the precise detection of corresponding Lidar and camera points. For this reason, a calibration target was deployed that combined the integration of an AprilTag visual fiducial mark  along wiRegarding the Lidar marker, two reflective crossing stripes were attached to a planar board, while at their intersection point, a small AprilTag marker was carefully placed, ensuring the precise alignment of both center points . We keptWe proved experimentally that such a small AprilTag size (a few centimeters), considering our 4K camera employed in the calibration procedure, could be automatically detected and precisely localized for a wide range of camera-to-target distances, reaching a maximum distance of 6 m, quite adequate for our sensor configuration.Like the image marker, the size of the Lidar marker (planar board and stripes dimensions) is also irrelevant to the calibration procedure. Its center is accurately estimated from the two crossing stripes, provided that each stripe is covered by several successive laser points (min 2–3 points) across and several laser rows (min 3–4 Lidar channels) along its length. In the example in The detection of the Lidar target was carried out automatically due to the distinctiveness of the retroreflective stripes. Setting a minimum threshold  for the recorded intensity of all laser beam returns, an initial clustering was carried out, which effectively eliminates all non-target points (low reflectivity measurements). In In the second step, the equation of the board plane was estimated using a least-squares minimization approach for all target coplanar points. The employment of only the retroreflective 3D points in the estimation of the board plane equation proved to be more accurate than exploiting all Lidar points on the board  since the highly reflective materials increased the precision of the range measurements. The target’s 3D points were projected perpendicularly onto the estimated plane and served as the basis for the remaining 2D calculations.Subsequently, applying the RANSAC  method iIn The accuracy of the extracted target center may be improved by merging several (2~3) successive static scans of the target, provided that the target remains stationary during the capturing phase. Using this technique, the precision of the estimated target center may be further increased, yielding a more accurate alignment between the Lidar and camera measurements.The result of the feature extraction stage was the establishment of accurate 3D−2D point correspondences between the Lidar and the camera reference system from successive measurements of the proposed calibration target from both sensors. These measurements were the inputs of the geometrical and temporal calibration procedures.In the preliminary stage, the internal parameters of the camera should be accurately estimated. This procedure can be carried out by utilizing open-source toolboxes, such as the OpenCV camera calibration for ROS , the MatUtilizing the current feature extraction technique, a sufficient number of point correspondences between 3D Lidar and 2D image points needs to be established for a reliable and accurate exterior calibration. In the first step, the camera exterior orientation is initialized with respect to the Lidar world system by a linear PnP (Perspective-n-Point) ,30 soluti, i, K is the matrix containing the camera intrinsic parameters, and iR and it are the rotation matrix and translation vector, respectively [The 6 estimated extrinsic parameters (3 rotations and 3 translations) are further refined by applying a non-linear least-squares PnP minimization. The cost function that is minimized (Equation (1)) estimates the L2 Norm (Euclidean distance) of the errors between extracted 2D image points and the projected 3D Lidar points in the image (reprojection error):Following the geometrical calibration procedure, a temporal calibration is applied to address synchronization issues between the camera and Lidar recordings. This is a severe problem that occurs, especially in camera–Lidar systems that lack hardware synchronization .i (AprilTag marker) and i should coincide  with image point i all 3D target points t), the temporal calibration algorithm selects the projection j to the current image frame i.The algorithm described here attempts to match the 2D location of an extracted AprilTag fiducial from a specific image frame to the closest 2D image location of several successive Lidar target points after being projected onto the specific image. Let This implies that the calibration target should be moving at a relatively constant speed during data capturing to prevent possible temporal misalignment. Apart from the constant velocity of successive target detections (target tracking), another criterion is enforced here that concerns the direction of the target’s movement. According to this, an angular threshold bounds the direction of every new detected target within an acceptable angular range with respect to the direction of the preceded target. Thus, the possibility of mismatching images to Lidar targets is further reduced.Combining several such target measurements from different times during the acquisition, the algorithm can accurately infer the overall time-shift of the camera with respect to the Lidar recording.Our calibration algorithm is provided as an open-source implementation in the ROS framework and is primarily aimed at non-expert users by keeping the calibration approach as simple and effective as possible. In addition, the high accuracy achieved in locating the target in both sensing systems (camera and Lidar) can establish the proposed algorithm as a key calibration tool, not only for research but also for production purposes, allowing it to be also used during the evaluation similar calibration approaches see .The ROS  platformFor our calibration toolbox, the input data should be compatible with the requirements of the ROS framework. Two different topics are required: one for imagery containing camera messages (uncompressed image data) and one for 3D scans containing messages from the Lidar (point cloud messages). The internal parameters of the camera are fed to the application using a camera info topic or as an external file containing a camera matrix and distortion coefficients. It is also possible for the user to store the topics in a rosbag file and perform the calibration offline.Regarding geometrical calibration, first, the software takes successive target measurements while the user moves the calibration target to various places. For increased calibration accuracy, the ‘active area’ corresponding to the overlapping FOV of both sensors ought to be covered by an adequate number of equally spaced target measurements. To this end, the algorithm controls the target’s density by considering only measurements that satisfy the minimum distance criterion to each other. Moreover, from the displayed locations of all previous measurements , the useAiming to achieve reliable results, even in configurations with significant time delays among camera and Lidar recordings, we utilized an image tracking method to eliminate abrupt target movements. Thus, the algorithm can detect slow-moving sequences of the target in real-time and take a measurement only when the target is practically stationary. This is achieved by the parameter “distance threshold”, which concerns the minimum displacement from frame to frame, and parameter “consequent frames”, which determine the maximum number of consequent frames being taken into consideration . This caAfter the capturing phase, the users can visualize the calibration results in the command window and the spatial distribution of the residual (reprojection) errors displayed on the image for every observation individually. In In addition to the geometrical calibration, the user, exploiting a similar interface, can perform temporal calibration in order to synchronize the camera to the Lidar frames. In this case, and in contrast to the previous approach, the distance parameter is applied here as a minimum ‘distance threshold’ among consecutive target observations, which aims to prevent the measurement of slow-moving targets. A target observation is considered valid only if the minimum distance criterion is met for an adequate number of concurrent frames (the ‘consequent frames’ parameter).Since the geometrical calibration is based on the accurate synchronization of the recordings captured by the camera and the Lidar sensor, it is advisable, especially for datasets with larger synchronization differences, to repeat the geometrical calibration after the synchronization step.In this section, we present experimental results to evaluate the accuracy of our calibration methodology as well as the effectiveness and flexibility of the proposed ROS implementation. Our software was exploited for the calibration of a custom-made mobile mapping platform consisting of four embedded 13MP machine vision cameras (by e-con Systems) with respect to a Velodyne PUC VLP-16 Lidar sensor mounted on top of the cameras . A thoroSince the current configuration of the MMS does not allow any image overlap between the four cameras, the procedure is mainly based on the independent calibration of each camera with reference to the Lidar sensor. First, the interior orientation of each camera was estimated using fauccal . After aThe results are quite satisfactory given primarily the modest pixel error of the adjusted observations. It was also observed that the parameters of the four camera-to-Lidar transformations were noticeably consistent and apparently close to the real relative positions and rotations of the sensors. This fact also holds in For the purposes of comparison, the developed calibration approach was also evaluated against the results from a rigorous open-source calibration toolbox commonly adopted by the mobile mapping community . Eleven For a more reliable assessment of both calibration results, a set of well-defined 3D points was projected onto the camera frame using the external parameters yielded by both solutions independently (our method and Zhou’s method). The 3D points were extracted by moving our proposed calibration target into different positions while it was being recorded by the camera and the Lidar device. The 3D target centers extracted from the Lidar scans were projected onto the image and compared against the corresponding 2D target centers extracted on the image, giving the final RMSE for both datasets.The deviations and the computed RMSE of the 22 measurements for the two approaches are shown in However, the difference between the two is considerably greater if one considers the convenience of the calibration process. In the case of Zhou et al.’s method, and according to our experience with the corresponding Matlab 2022a implementation, there are a lot of prerequisites one should consider for reliable calibration. First, the scans of the chessboard plate should not be disturbed by any obstacle, particularly the user’s fingers holding the plate, which is a very common case. Similarly, the user should not keep the plate too close to his body. Both cases can significantly deteriorate the calibration results by intervening either during the edge extraction of the 3D chessboard (the former case) or in the determination of the planar board equation (the latter case). Our approach can directly overcome these barriers, keeping only the structural information of the reflective target points. Additionally, and apart from the requirement for tilted checkerboards during the acquisition phase, the checkerboard-based algorithm did not perform satisfactorily when detecting the board planes from the successive scans, especially in indoor environments with many planar objects present. In several 3D scans, user interaction was inevitable. On the other hand, the proposed calibration technique does not require any a priori structural information for the calibration target, such as checkerboard square size or padding dimensions, keeping the workflow as intuitive and flexible as possible for the user. The only requirement is the creation of the current calibration object with two reflective stripes in a crossing layout, which is a rather simple and low-cost endeavor.Finally, the calibrated relative rotations and translations between the four cameras (camera rig) and the Lidar sensor of our mobile mapping system were assessed during a SLAM and a texture mapping procedure. A final 3D map was generated employing the Lidar-based SLAM approach of the RTAB-Map package  from 10,In this contribution, we present a novel approach for automatic camera-to-Lidar extrinsic and temporal calibration based on an easy-to-create calibration target. For the geometrical (extrinsic) calibration, the 3D rigid transformation of the camera system is estimated with respect to the Lidar frame on the basis of the establishment of 2D to 3D point correspondences. The temporal calibration (sensor synchronization) is carried out by matching the position of the calibration target detected on the image frame to the corresponding 3D target projected on the same. The geometrical calibration was evaluated against a state-of-the-art approach, and it yielded very satisfactory results. Both geometrical and temporal calibration can be carried out by the user through our open-source implementation in ROS. The effectiveness and reliability of the current approach make this toolbox suitable for production purposes and for researchers during the evaluation of their own calibration algorithms. In the future, we plan to integrate a more rigorous model for the plane-fitting step and to further evaluate the proposed calibration method with measurements from various cameras and Lidar sensors. Finally, we intend to make our toolbox available for Matlab users.


This paper presents a system for behavioral, environmental, and physiological monitoring of working dogs using on-body and aerial sensors. The proof of concept study presented here includes two trained dogs performing nine scent detection tasks in an uncontrolled environment encompassing approximately two acres. The dogs were outfitted with a custom designed wearable harness to monitor their heart rate, activity levels and skin temperature. We utilized a commercially available micro-air vehicle to perform aerial sensing by tracking the terrain and movement of the dog in the outdoor space. The dogs were free to explore the space working at maximal speeds to complete a scent-based search-and-retrieval task. Throughout the experiment, the harness data was transferred to a base station via Wi-Fi in real-time. In this work, we also focused on testing the performance of a custom 3D electrode with application specific ergonomic improvements and adaptive filter processing techniques to recover as much electrocardiography data as possible during high intensity motion activity. We were able to recover and use 84% of the collected data where we observed a trend of heart rate generally increasing immediately after successful target localization. For tracking the dogs in the aerial video footage, we applied a state-of-the-art deep learning algorithm designed for online object tracking. Both qualitative and quantitative tracking results are very promising. This study presents an initial effort towards deployment of on-body and aerial sensors to monitor the working dogs and their environments during scent detection and search and rescue tasks in order to ensure their welfare, enable novel dog-machine interfaces, and allow for higher success rate of remote and automated task performance. Dogs play a critical role in modern society and perform an array of functions from simple quality of life interactions such as companionship, to professionally trained operations such as herding livestock; detecting drugs, explosives, and pests; and aiding in search and rescue (SAR) applications. A number of studies in the literature show that monitoring physiological signals  and behavioral indicators  not only provide insight into the physical and emotional health and welfare of the dogs, but also allow for computer-assisted training approaches ,4,5,6,7.Commercial unmanned air vehicle (UAV)-based surveillance has great utility in SAR applications for recording the search activity and surveying the environment. This is especially useful in relatively harsh environments that are challenging for human exploration but dogs can work relatively easily in, such as rubble piles after natural disasters or large agriculture fields and forests. In these environments, dogs can be excellent trackers on the ground where UAV cameras in the air allow for situational awareness and further insight into the detection tasks.In this paper, we present our efforts towards combining the visual, environmental, and navigational sensors carried by a commercial UAV with a custom designed sensor-equipped canine harness ,15,16. WThe system described in this paper affords the first attempt towards fusing on-body and aerial sensors for multimodal interrogation of working dogs’ performance and wellbeing through data analytics that incorporate streams of physiological, behavioral and environmental data. We describe preliminary steps taken towards this goal and provide experimental results for proof of concept validation of sensing and system performance in a realistic wilderness environment. Analytical interpretation of sensor data reveals promising results, demonstrating that the sensing systems can work effectively in challenging environments and during high-level activity of the dogs.In our prior work, we examined the benefits and limitations of various on- and off-body sensing of behavioral, environmental and physiological parameters for working dog applications ,21,22,23Smart America Challenge in 2013, organized by the White House in which a Smart Emergency Response System (SERS) was developed to connect cyber-physical technologies (robots and drones) with humans and instrumented dogs to help save lives in disaster areas [There has been a recent interest in enabling dog-drone or -ground robot interactions, especially for SAR applications. Previous research shows that dogs can be trained to follow a UAV or receive commands in such scenarios and engage in functional referential communication ,26,27,28er areas ,11. In ter areas ,30.Synergistic Interactions in Swarms of Heterogeneous Agents (SWARMIX) [Another major effort to combine UAVs and working canines was a notable computer-networking-focused project, based in Europe, SWARMIX) . The proThe details of our harness system can be found in . This sy the dog . For thiAn 85 g 5000 mAh battery powered the system and this can be easily extended as dogs can carry much larger payloads on the order of several kilograms. In practice, we achieved up to 12 h of continuous data transmission of the system with the given battery. The twelve hours of transmission time far exceeds the working time an individual dog would be required to perform a scent detection task. In search and rescue scenarios, handler-dog teams take regular breaks throughout a deployment. While deployments may last days or weeks and teams may work 8–12 h per day, seldom would any team work more than a two hours without a substantial rest period. At this point, the dog would be given a break, the equipment would be removed and a fresh battery would be used for the dog’s next activity.For this study, we distributed a set of temperature sensors DSB18B20  across the harness to collect skin versus ambient temperature as a vital sign. Three waterproof DS18B20 temperature sensors were added onto the board via terminal blocks and digital inputs with a sampling rate of 0.5 Hz since temperature is expected to change slowly. One sensor was placed between the strap of the harness and against the chest of the dog to monitor skin temperature. A second temperature sensor was placed inside the electronic pouch environment which houses the electronics. Finally, a temperature sensor was placed to protrude out from under the harness to measure the ambient temperature in the environment. This allowed for a complete picture of the micro-environment of the dog during search activities. In addition, the temperature of the Raspberry Pi central processing unit (CPU) was monitored as previously we had seen performance issues when the CPU temperature increased too high, when the dog is working under sun or hot environments.While it is possible to perform some amount of streaming data analysis on a Raspberry Pi microprocessor, doing so limits the ability to synchronize analysis with other streams of data collected from other sources . Thus, for this article our approach was to leverage the Raspberry Pi primarily as the microcontroller to collect and transmit raw data to a central location where it could be synchronized with other sources.As such, A modified version of the “CanineEvaluation” software  served aA commercially available DJI Mavic II Zoom  was flown by an experienced pilot and used to film and track the dog throughout the experiment. This drone has a 4K resolution camera and is capable of speeds up to 72 km/h and flight time of over 30 min. We synchronized the data streams of the drone video with the ECG, IMU, and temperature sensors by manual time markers applied at the beginning and end of the experiments. The experiments took place in an area of approximately two acres. We used a Rocket M2 2.4 GHz CPE AirMax router and AM-2G15-120 AirMax Sector 15 dBi 120 deg antenna . The hardware is specifically designed for harsh weather and outdoor environments, which makes it ideal for this type of experimentation. Prior to experimentation, we walked the boundary of the search area by venturing more than 150 m from the router without line of sight due to tree obstructions and did not lose signal. This allowed us to transmit the ECG and IMU data back to the base station in real time throughout the search area.All animal procedures were approved by the NC State University Institutional Animal Care and Use Committee (IACUC). We ran four experiments with two male Labrador Retrievers (four and eight years old). We had nine search-and-retrievals in an area of approximately two acres. We outfitted the dogs with the harness and hid typical toys, called “bumpers”, used in retrieval training. We used positive reinforcement operant conditioning to train the dogs to search the bumpers. The field was set up to allow the dog to search for bumpers safely. The handler walked and occasionally pointed the dog during the search, and retrievals were noted in the software. The two dogs had previously been acclimated to performing tasks while under drone surveillance and not affected by its operation.Three axes of the IMU provided an activity level vector. The vector was used to detect excessive motion and reduce motion artifact in the ECG signal. The activity level was averaged over 5-s windows. The activity and heart rate were examined 10 s before and after retrieval. In addition, the activity vector was used to perform adaptive noise cancelling in areas of excessive motion to provide more accurate heart rate estimation. The ECG system and our peak detection algorithm was previously validated against commercial devices for resting and average intensity level activities such as walking at a lower speed and regular pace . Our effTo demonstrate the functionality of the aerial camera sensing, we applied a state-of-the-art online single-object deep tracker, the SiamRPN method , to trac(i)Initializing the bounding box of an object instance to be tracked in a given frame of a video: Based on the initialization, a target model is learned (see (ii)Tracking the object in the subsequent frames: For improved efficiency, a local search region is defined in the current frame based on the tracking result in the immediately previous frame (see (iii)Updating the target model based on the new tracking results: Since the initial target model is learned with a single observation of the object of interest in tracking, it is desirable to update the target model with more observations obtained to account for structural and appearance variations of the object of interest. Thanks to the expressive power of DNNs, the SiamRPN method enables us to obtain reasonably good tracking performance on the challenging drone videos.This online tracking algorithm has the following three components:normalized precision between the tracking results and the annotated dog bounding boxes, which are summarized as follows.In evaluating tracking performance, we annotated dogs in each frame of two videos with bounding boxes using the free and open source “labelImg” tool . For a gthe success rate of tracking, the intersection over union (IoU) between the ground-truth box and the prediction box is calculated in each frame, and then the success plot is generated by varying the IoU thresholds from 0.05 to 1.0 (inclusive) with a step size 0.05. For a given IoU threshold, the tracking is said to have succeeded in a frame if the IoU between the ground-truth box and the tracking result is greater than or equal to the given threshold. The success rate for a given threshold is the proportion of frames with successful tracking results. The mean success rate is the average over all the IoU thresholds.To characterize the precision of tracking, the distance between the center of the ground-truth box and the center of the prediction box is calculated. In our case, the distance threshold varies between 0 and 51 pixels with a step size of 1. For a given threshold, the precision rate is defined by the proportion of frames with the center distances smaller than or equal to the threshold. The mean precision rate is the average over all thresholds. The normalized precision of tracking is calculated using both the center of ground-truth boxes and the center of prediction boxes, and normalizing those values by dividing each according to the side length of ground-truth boxes.To characterize the precision rate only evaluate the relative distance between the ground-truth and the prediction without checking the shape and size of the boxes. The success rate captures both the distance and the shape and size.Intuitively, Video files from the UAV camera were archived on local storage to a central repository in the cloud before backup for subsequent analysis during training of the SiamRPN++ model. It should be noted that, although not implemented here, the SiamRPN++ model, when doing inference as opposed to training, can run on a suitably equipped mobile device . However, while training, the requisite compute resources are more significant and thus the workload is better performed in the cloud.We set out to measure behavioral, physiological, and environmental data using both on-body harness and off-body UAV based aerial sensing in dogs during scent detection tasks.We used ECG waveforms to extract heart rate based on R peak detection. Initially, we extracted heart rate from the ECG signal using the “find peaks” function in MATLAB where peaks are detected with a minimum local maximum of three times the average signal value and minimum gap of 143 ms per peak. The initial analysis showed exceptionally high heart rates during intense activity above the expected maximum, as a result of motion artifacts. The observed instances of heart rate in the order of 360 beats per minute (bpm) is well above the expected physiological maximum of around 300 bpm . This caIn order to improve heart rate estimation, the three axes of the IMU are vectorized:While motion artifacts can be mostly addressed through ergonomics of electrode design and signal processing, the trend of the data collected indicates a potential additional improvement through careful design of the analog front end circuits by adding an adaptive gain stage in the future. The data collected in the field experiments suggest that the amplitude of the ECG signal is the highest at points of highest motion and as such, there’s a proportional relationship between the activity level and peak magnitude of the ECG signal. An adaptive amplifier could be utilized to perform amplification at an inverse relationship to the activity level, decreasing the gain when motion is high and increasing the gain during lower activity. This would improve the overall signal to noise ratio of the ECG signal.In all, we ran four experiments with two different dogs performing two experiments each with a total of nine object retrievals. We examined the interplay between heart rate and activity level 10 s before and after for all nine retrievals as shown in While our aim here is to provide proof-of-concept data about the use of on-body and aerial sensors for tracking dogs, we note that our observations lead to interesting trends with a need for further investigation in the future, especially to explore the relationship between the increase in heart rate associated with the dog after object retrieval despite a decrease in activity level. The decrease in activity level generally occurred because the dog returned the bumper to the handler; however, it is interesting to note that in seven of the nine retrievals the heart rate starts to decrease as the bumper is retrieved and increases after retrieval. We keep this investigation beyond the scope of this article and reserve for a future study.The temperature sensors distributed across various points on the harness also revealed interesting results. We collected the temperature inside the electronics pouch, the CPU temperature, the ambient temperature, and the temperature between the dog’s fur and the computer as shown in normalized precision performance were achieved in the scope of this study. Two other representative image examples are provided in In this paper, we present the first study fusing on-body wearable sensors and aerial-visual sensors to monitor working dogs. We particularly focus on offering solutions towards assessment of two important signals while the dog is performing a high speed scent-detection task in a large area. These signals include the heart rate acquired with the on-body ECG electrodes and macro-level activity of the dog using visual sensors. In this section, we would like to widen the scope and offer a broader vision that can be achieved with such stream of dataflow acquired from a suite of multimodal on-body and aerial sensors in real time. This novel data stream could be essential to determine the true performance of the working dog, characterize confidence in searches, improve situational awareness, and optimize search patterns to increase effectiveness while also ensuring welfare of working dogs.Due to space limitation, several aerial-sensor enabled macro-environment analysis is kept beyond the scope of this paper, including UAV cameras examining the topology of the land to identify certain characteristics that would impact search performance . The fusion of aerial UAV cameras and on-body IMUs and global positioning systems (GPS) could offer very high resolution tracking of the dog throughout the search whether it is on- or off-sight of the visual sensors. Beyond providing a live video feed of the surveyed area, the UAVs can also be used as wireless data relays to transmit any information received from electronics carried by humans or dogs to distant base stations or directly to the Cloud ,9. The tThe combined on-body and aerial sensing of the working dog and the environment could also comprise a basis for novel in vivo psycho-physiological studies of working canine behavior as well as avenues for advanced algorithmic analysis and optimization of handler-canine-drone triad performance in real-world settings. Working canines are among the most sophisticated mobile olfactory sensors in existence, and it is well documented that air conditions  heavily influence the performance of these working animals. For example, the way air swirls up and over an obstacle can lead to cyclic scent flows emanating from the obstacle. Alternatively, on a warm, high-humidity day a dog may generate excessive body heat in low-lying terrain where airflow is stalled by vegetation, thereby reducing the dog’s olfactory effectiveness. The tracking of micro- and macro-level factors with environmental sensors placed on harness and aerial visual sensors  could provide a better assessment of the dogs working conditions and effect of these on the its success rate for scent detection tasks. Moreover, combination these with tracking of the heart rate, respiratory patterns and movement of the body could further improve this success rate ,15,17.The system presented in this study with the addition of other commercially available sensors and more advanced data analytics hold promise for such a grand vision that will be the focus of our future work on this topic.egocentric and allocentric data representing canine performance and welfare, as well as location within the environment and interaction with specific objects. This capability goes beyond simply using GPS for geolocation to enable a system whereby canine behavior can be analyzed in concert with environmental factors discernible by drone-based environmental sensing.In this paper, we used on-body and aerial sensing to monitor the behavior, physiology, and environment of a dog during search and rescue tasks. We successfully measured heart rate, activity level, and temperature via on-body sensors placed on a wearable harness. We used commercial UAVs with cameras for aerial tracking of the dog throughout the experiment. Taken together, these two categories of capability substantively increase the ability to monitor and characterize the performance of working dogs in vivo. Wearable and drone-based sensing combined enable simultaneous collection of both We particularly focused on design enhancements to address experimental challenges imposed by high intensity motion activity of dogs performing scent detection tasks in uncontrolled environments. We achieved a robust wireless network encompassing a nearly two acre space and allowing for real-time data streaming. In this space, our proof of concept experiments included two dogs and nine search-and-retrieval sessions. It is very challenging to acquire ECG signals during excessive motion of working dogs. We improved and tested the reliability and robustness of an application-specific 3D printed electrode during maximal working speeds. A custom designed leather harness improved the ergonomic fit and proved rugged enough to withstand the uncontrolled, outdoor environment. The heart rate estimation was improved through adaptive filtering by using the activity level from the IMU to remove excessive motion during the bumper retrieval. Signal processing techniques recovered 84% of the data in an application where it was not possible to collect usable data previously. In general, we saw the dog’s heart rate increase upon retrieval of the bumper and recover after the retrieval was complete and it waited to be released.Using deep neural networks applied to UAV visual recordings for object tracking, we accurately tracked dogs’ movements throughout a two acre field. With this, we performed automated video tracking of dogs in the wild and obtained promising results. We analyzed the barriers towards improving the tracking performance further and offered suggestions for a fully automatic tracking algorithm by integrating DNN object detection and online object tracking in the future.While many challenges exist to realize the full capabilities of automated canine-human-drone triads, the work presented in this paper represents a significant effort forward in overcoming some of the most significant ones. Fusing data streams obtained from on-body and aerial sensors could enable more efficient performance of remote scent detection search tasks and ensure the welfare of the working dog during potentially dangerous environments.


Recent research trends focus on developing bio-based  fiber-reinforced polymer (FRP) composites for structural applications. Fire resistance is one of the key issues that need to be addressed for the use of these FRP materials in buildings. The thermal and mechanical properties of the constituent materials essentially determine the fire performance (and the fire resistance rating) of a structural member, and these properties vary with temperature. Further, the properties of composite materials such as the FRP are highly influenced by the composition and type of fibers and matrix, and these thermo-mechanical properties also vary significantly with temperature. Due to this variation, the fire resistance of FRP materials  poses a major concern for use in buildings. Currently, very few standardized test procedures are available for evaluating the high-temperature material properties of FRP composites. In this paper, a review of testing protocols and procedures for undertaking tests on FRP materials at various elevated temperatures for evaluating their properties is carried out. Recommendations are provided on the most suitable test methods, specimen conditions, testing regime, and other issues associated with testing at elevated temperatures. In addition, the applicability of the proposed test methods is illustrated through a case study on conventional FRP specimens. Further, the applicability of the recommended test procedures for measuring high-temperature properties of bio-based FRP composites is highlighted. CurrentNatural fibers are being considered as a potential alternative, in place of synthetic fibers, for fabricating bio-based composite materials. Biomass converted to ceramic nanoparticle in forced cellulosic fibers and combined with agricultural side products  is being processed to prepare the bio-based FRP composites. Typically, plant-based fibers are derived from agricultural waste/stubble, and this alternative is gaining considerable interest owing to their superior advantages over synthetic fiber, such as their relatively low weight, locally available raw ingredients, low cost, eco-friendly, and acceptable strength and stiffness properties. Such bio-based FRPs can be utilized in buildings in the form of fibers, rebars, strips, sandwich panels, and other decorative face sheets to achieve energy savings and economy in the construction sector. In addition, the overall carbon footprint resulting from the construction sector can be reduced.Tg), which affects the force transfer between the structural member and the FRP reinforcement [Some of the major issues with the FRP composites, including bio-based FRPs that limit their application in buildings, are their high flammability and poor fire resistance properties. The FRP components are highly combustible and decompose even at low to moderate temperatures in a fire situation releasing soot, smoke, and toxic gases. This leads to the generation of toxic gases and poor visibility conditions, which can hinder the occupant evacuation process and firefighting operations in a building. From a structural perspective, the FRP composites rapidly lose strength and stiffness at moderate temperatures of 200 °C to 300 °C due to the degradation in the properties of the polymer matrix with temperature rise . MoreoveStructural members in buildings are required to meet the fire resistance ratings prescribed codes and standards, in addition to other criteria such as flame spread ratings, smoke generation classifications, etc. For evaluating the fire resistance of an FRP-strengthened concrete structural member, information regarding the thermal and mechanical properties of the FRP composites together with that of concrete and steel reinforcement are needed. The thermal properties determine the thermal behavior of the composite material  including the extent of temperature rise in a member. Mechanical properties, which include strength, modulus, creep, and bond strength, determine the loss of structural capacity, and the extent of deformation in a member with fire exposure time. The properties of the FRP vary significantly with temperature compared to that of conventional construction materials and need to be reliably measured through standardized test methods.The material properties of conventional FRP have been studied widely at ambient temperature conditions ,3,4. In As mentioned earlier, there are many merits to synthetic fiber-based composites with certain disadvantages too, such as difficulty associated with the disposal of synthetic fibers and high cost. Currently, researchers are focusing on natural fiber-reinforced composites for building applications in the fabrication of structural and non-structural elements, and various other infrastructure systems especially where sustainable materials of low weight and moderate strength are required. For the past few decades, natural fiber composites have been used in the automobile industry for making various body parts, rotor blades, and lining materials. In the infrastructure market, bio-based FRPs are utilized in making water tanks, load-carrying beams and columns, pedestrian footpaths, and panels for multi-purpose use .Natural fibers are also used to improve the performance of the composite through hybridization with synthetic fibers. Such hybridization reduces moisture absorption, in addition to balancing the cost of the materials and allowing for surface modification using chemicals . The hybNatural fiber-based sandwich composites are also being used in the fabrication of flexural members such as beams, bridge decks, and girders . TicoaluCurrently, to prevent the deterioration of reinforced concrete structures due to aggressive environmental conditions, natural–synthetic fiber-based hybrid composites are also being used by wrapping these composites over the exposed surface of concrete structures. For example, the wrapping of hybrid sisal glass fiber-reinforced composite polymer (HSGFRP) around the concrete cylinder has improved the performance of the test specimen . FurtherThus, from the above literature, it is observed that natural fiber-based composites are used as a reinforcing material globally due to their inherent advantages and have a wide variety of applications in many sectors where synthetic fibers have been used so far. Most importantly, an attempt is made by researchers globally to extract the natural fibers from agricultural waste and use them to develop natural fiber-based fiber-reinforced polymers for different structural applications and, in turn, contribute to the growth of national gross domestic product (GDP) and generate the agro-based economy and its employment opportunities.The material property data for bio-based FRP are limited to room temperature conditions and also mainly to certain mechanical properties of the FRP. As seen from Temperature-dependent properties of FRP are essential in addition to other constituting materials for evaluating the fire performance of FRP structural members. The key properties are discussed below, in detail.Flammability and smoke density properties determine the flame spread and toxicity of a material when used in building applications. These properties do not have a direct relationship with the fire resistance  of an FRP composite member. However, these properties are of importance from a fire safety point of view since a high rate of flame spread and toxic smoke development pose a severe hazard to the occupants of a building. For this reason, structural members in buildings must meet certain flame spread index (FSI) and smoke density index (SDI) requirements as prescribed in current codes. When the FRP is incorporated into a structural member, the FSI and SDI ratings are altered due to the combustible nature of the FRP. Unlike steel and concrete, which are non-combustible, the FRP is a combustible material and thus is flammable in a fire scenario. The material composition of the FRP composite largely determines the flame spread and generation of toxic smoke when exposed to fire.Tg). The thermal conductivity and specific heat of the FRP composite have a direct influence on the rate of temperature rise within the structural member, and hence, on the fire resistance of an FRP composite member. The glass transition temperature indicates the temperature at which the polymer matrix turns to a viscous state leading to a drastic reduction in the strength and stiffness properties of the FRP composite, which in turn can severely impact the fire resistance  of an FRP composite member. Hence, the glass transition temperature has a direct relationship with the fire resistance of an FRP member. Finally, the thermal expansion of the FRP composite influences the stresses and deformation response of an FRP composite member, which in turn can directly affect the fire resistance of an FRP composite member. Depending on the type and amount of the FRP used, the level of influence that the thermal properties have on the fire response varies. For instance, in standalone FRP components and concrete members wrapped externally with the FRP laminates, the FRP composites are exposed to fire directly and thus, can affect the heat transfer within the member. In this case, the thermal properties of the FRP cannot be neglected in the fire resistance evaluation. However, when the FRP composite is used as an internal reinforcement within a concrete member, the thermal properties of the FRP tend to have negligible influence owing to the smaller cross-sectional area of the FRP reinforcement, as compared to that of the concrete section. In this case, the FRP reinforcement can be considered similar to that of steel rebar in the thermal analysis [The thermal properties of the FRP that are of interest from the perspective of fire performance of a structural member include thermal conductivity, specific heat, thermal expansion, and glass transition temperature  fibers, depending on the fiber composition, preliminary studies have shown that they are only able to retain a small part of their strength in the 180 °C to 400 °C range . In bothAnother mechanical property that significantly influences the fire response of the FRP-strengthened members is the bond strength. Sufficient bond strength is required between fibers and matrix and as well as the FRP composite and concrete to facilitate the transfer of stresses between the matrix and fibers and between the concrete member and the FRP system and vice-versa. However, due to the vulnerability of matrix performance to elevated temperatures, the bond properties of the FRP undergo significant degradation even at a moderate rise in temperature of 80 °C , which cYet another mechanical property that influences the fire response of the FRP components is the creep strain. In general, creep strain in the FRP increases with an increase in temperature and is largely governed by the type of polymer, type of fiber, and orientation of fibers in the FRP composite. Creep strains are extremely difficult to measure at elevated temperatures, and the degradation of the polymer matrix makes it even harder to measure this property of the FRP at high temperatures. Therefore, very limited information is available on the high-temperature creep strain of the conventional FRP. Since the FRP area is much smaller than that of concrete in the overall strengthened member, creep in the FRP is often neglected in fire resistance analysis .The above-described mechanical properties have a direct relationship with the fire resistance of the FRP composite member. The higher the tensile strength, stiffness, and bond strength of the FRP composite, the higher the load-carrying capacity and lower the deformations in the member at elevated temperatures, and hence, the higher the fire resistance of the member. The test methods, conditions, and procedures for measuring the various flammability, smoke density, thermal, and mechanical properties of FRP composites are discussed below.Standardized test methods are recommended in the ASTM and National Fire Protection Association (NFPA) standards to evaluate the flame spread and smoke generation of structural materials. The relative burning behavior of building materials can be evaluated by measuring the FSI and the SDI through procedures specified in ASTM E84  and NFPAThe thermal properties of an FRP composite can vary significantly based on the test methods, specimen conditions (preparation), and the type of instrument (or equipment) used to measure these properties. This high variation is because unlike steel and concrete there can be a high variation in the fiber-matrix composition and also the instability of these composites even at moderate temperatures 100–300 °C). For measuring thermal properties of construction materials at room temperature, different test methods are specified in the ASTM standards such as ASTM C177 , ASTM C10 °C. ForThe procedure to measure the thermal conductivity of the conventional FRP composites at ambient conditions is given in ASTM C177  and ASTMThe ISO test standard 22007-2  specifieFrom the different methods discussed above, it is recommended to use the hot disc method specified in ISO 22007-2 to evaluate the thermal conductivity of FRP specimens  in the temperature range of 20–300 °C. A Kapton TPS sensor of radius 6.4 mm, a test specimen of size 50 mm square (on each side), and 25 mm in thickness are recommended for testing. The error in thermal conductivity measurements at higher temperatures using the ISO 22007-2 is around ±5% of the true value . HoweverSpecific heat of conventional FRP composites can be measured using the differential scanning calorimetry (DSC) method specified in ASTM E1269  and ISO ISO 22007-2  recommenStandardized test methods for evaluating the thermal expansion of construction materials are specified in ASTM E831-19  and ISO As per the provisions in ISO 11359-2  standardTg) of FRP composite materials is given in ASTM E1545-11 [The test procedure for measuring the glass transition temperature  is to be captured and for this purpose, a penetration probe is to be used. A constant heating rate of 2 °C per minute is to be selected and the temperature range of the test is restricted between 20 to 300 °C [The mechanical properties at high temperatures are highly sensitive to the testing regime (steady-state or transient state), specimen conditions , loading parameters , and the instrumentation used for the tests. Test procedures for evaluating the mechanical properties of conventional FRP at ambient conditions are specified in the ASTM standards such as ASTM D7205/D7205M-06  for tensDue to the lack of specifications for standardized test methods in standards, a review of experimental methods adopted by researchers in prior studies for evaluating the tensile strength and stiffness properties of FRP was undertaken. High-temperature material property data are available to a limited extent on FRP composites used in aerospace and automobile industries. However, such data are more scarce in the case of FRP composites used in construction applications . MoreoveBased on the available published information, the test method adopted by Yu and Kodur  to evaluIn a tension test, the FRP specimens are susceptible to crushing under the pressure applied due to gripping. Strong anchors are to be provided at the two ends of the FRP test specimen to ensure that the failure of the specimen occurs in the central region experiencing high temperature, rather than at cooler ends (or anchorage zone). Anchor systems in FRP coupons play a critical role in the test results  and for this purpose anchor system is to be provided as per the ACI 440.3  specificTo confine the filling materials, steel pipes with dimensions per ACI 440.3  specificFor the room temperature strength tests, the hydraulic Materials Test System (MTS) is used , which is to be attached between the upper and lower clamping brackets. The loading is incremented until the specimen elongates considerably and fails. Data from the tension tests carried out on the FRP specimens can be used to evaluate the tensile strength, elastic modulus, and stress–strain relations at various temperatures. The tensile strength at a tested temperature is taken as the maximum load at failure divided by the actual cross-sectional area of the test specimen, whereas the slope of the linear portion of the stress–strain curve is taken as the elastic modulus. At each target temperature, at least two tension tests are to be conducted, and the average of the two (or more) values is taken as tensile strength and elastic modulus of the tested FRP specimen.The information on bond strength and bond-slip relations are critical for evaluating the capacity and deformations in an FRP component or FRP-reinforced concrete member. The bond strength of FRP can be measured using pull-out tests conducted at room temperature conditions. Although there are standardized test procedures for evaluating bond strength at ambient temperatures, no guidance is available in the current standards for high-temperature tests. Similar to strength tests, bond tests at elevated temperatures are quite complex due to issues such as gripping and softening, and burning of the FRP composite, specifically the matrix. In the literature, very few tests are reported at elevated temperatures, especially in the temperature range experienced by the FRP composite member during fire conditions. Of these methods, the test method by Yu and Kodur  to evaluFor preparing the specimens for bond strength test, concrete blocks of 150 × 150 × 400 mm are to be cast from a batch of premixed concrete. FRP specimens in the form of bars or strips can be tested using this method. One end of the specimen is to be bonded with the concrete block while the other end is to be installed with a strong anchor system that facilities gripping of the FRP specimen during the pull-out tests. The anchor system is to be designed as per the ACI 440.3  specificThe test equipment for carrying out the high-temperature bond tests is comprised of a tension-testing machine and an electric furnace to subject the specimen to elevated temperatures, as shown in Once the specimen attains the target temperature, the pull-out test is to be carried out by applying a tensile load using hydraulic jacks. The velocity of the loading is to be set at 2 mm/min. The slip between the FRP reinforcement and concrete block is measured using an LVDT that is to be placed between the top beam and the clamping brackets. The displacement of the top beam is to be continued until the FRP strip, rod, or plate is pulled out. The peak average bond stress  on the FRP composites are presented. These material property tests were carried out at various target temperatures on specimens made of the conventional FRP composites and following the recommended test methods discussed in the previous sections. Only the main results are presented in this section and detailed information on the test program and results are reported in other reports and thesis ,3,4,20.To illustrate the thermal expansion tests on FRP composites at elevated temperatures, tests carried out on four types of commercially available FRP reinforcement that are used in NSM strengthening applications are presented here. Measured dimensional changes in the transverse and longitudinal directions of the FRP specimens are shown in From the measured dimensional changes, the coefficient of thermal expansion at different temperatures is computed using Equation (1) and is tabulated in Tg) tests on FRP composites used in construction applications, high-temperature tests were carried out on Tyfo SCH-41 FRP specimen which comprised of unidirectional carbon fabric with glass cross fibers in an epoxy matrix [Tg value of 82 °C based on the ASTM D-4065 standard test dynamic mechanical analysis (DMA) [For illustrating glass transition temperature (y matrix . The ultis (DMA) . In thisTg, thus decreasing the signal-to-noise ratio and the transition temperature was shifted downward. This is illustrated in Tg and the average penetration of the probe drops [Tg of 64 °C is about 25% lower than the Tg reported by the manufacturer. This is mainly because the glass transition temperature provided by the manufacturer was determined by using a DMA, which depends not only on the heating rate but also depends on the frequency used in the test. Further, the varying atmospheric conditions and moisture content affect the glass transition temperature [Tg value reported by the manufacturer.be drops . Based operature , and theFor undertaking tests to evaluate the tensile strength, elastic modulus, and stress–strain relations of FRP at elevated temperatures, 25 CFRP specimens were utilized . In totaFrom the room temperature tests, the tensile strength and elastic modulus of NSM CFRP strips were found to be 1641 MPa and 150.8 GPa, while the corresponding values for the CFRP rods were found to be 1577 MPa and 130.9 GPa, respectively. While the elastic modulus of the CFRP specimens measured in the tests was comparable to those specified by the manufacturer, the room temperature tensile strengths were much lower than the strengths specified by the manufacturer. This is mainly because the manufacturer used only the strength of the carbon fibers in determining the tensile strength; however, in the tension test, CFRP (composite) specimens did not attain this strength due to the degradation (and fracture) of the epoxy resin [The reduction in elastic modulus and tensile strength of the CFRP strips with temperature is shown in To evaluate the bond properties of FRP composites, 36 pull-out tests were carried out on NSM FRP-strengthened concrete specimens at various temperatures . The speIn all the NSM FRP specimens, the bond strength and modulus degraded rapidly with an increase in temperature. The degradation of bond strength occurs at a faster pace in the 20 to 200 °C temperature range and at 200 °C, the NSM FRP retained only 20 to 30% of its original strength. The rapid degradation is attributed to the softening of the epoxy matrix beyond the glass transition temperature (around 80 °C) [The bond stress–slip relations at different temperatures for the different NSM FRP specimens are shown in As seen from the presented case studies; the recommended testing protocols are able to capture the variation of the thermal and mechanical properties of the FRP composite at elevated temperatures. Additionally, the recommended procedures for specimen preparation, fabrication of anchors (grips), and heating (and loading) regimes are practical and efficient for measuring the different properties of the FRP composite at elevated temperaturesThe recent focus in construction is to adopt bio-based FRP composites since it has numerous benefits including sustainability, environmental friendliness, low cost, locally sourced raw materials, etc. Temperature-dependent properties of FRP composites are critical for evaluating the fire resistance of standalone FRP components, and FRP-reinforced and FRP-strengthened concrete members. Preliminary studies clearly show that there is a larger variability in the properties of the FRP components made using natural fibers (bio-based) even at room temperature. At elevated temperatures, these plant-based FRP composites are susceptible to flaming, charring, and rapid loss of strength and bond properties, much like the conventional FRP composites. Moreover, the specimen conditions, such as the age of the specimen and moisture content, testing regime, and rates of loading and heating can vary significantly due to varying types of plants used, such as fiber, and this can have a high influence on the properties at ambient and elevated temperature conditions. For undertaking property evaluation, standardized test methods are critical for evaluating the properties of bio-based FRP composites.In the present study, a review of the various test methods adopted by researchers for the evaluation of thermal and mechanical properties of synthetic FRP composites is conducted. Upon detailed review, the most suitable test procedures for characterizing critical material properties of the FRP at high temperatures are recommended. The case study on conventional FRPs shows that the recommended test procedures can capture the variation in thermal and mechanical properties at elevated temperatures. Since the bio-based composites are of similar construction (comprising of fibers and resin) as that of the synthetic composites, the procedures and the recommended guidelines for specimen conditions, loading and heating rates, anchor systems, etc. can also be applied to undertake property tests on bio-based FRP composites for generating reliable material property data in the range of 20 °C to 600 °C. High-temperature property data compiled from these tests can be utilized to input relevant property relations in numerical models for predicting the performance of bio-based FRP under different fire exposure conditions.There is a lack of standardized test methods and procedures in current codes and standards for the characterization of high-temperature properties of synthetic and bio-based FRP in the temperature range of 20 °C to 600 °C.There are limited reported data on high-temperature properties and associated material property relations for conventional FRP composites and no such data are available for newer FRP composites incorporating bio-based FRP composite material.The recommended test procedures based on the review of the test methods in current standards and published literature can be adopted for the characterization of material properties of conventional and bio-based FRP composites in the temperature range of 20 °C to 600 °C.The case study shows that the recommended test procedures for the characterization of thermal and mechanical properties of FRP composites can be used to evaluate the high-temperature properties of the FRP composites reliably.A review of the test methods and procedures for evaluating thermal and mechanical properties of FRP composites at elevated temperatures is conducted and presented. Based on the reviewed literature, recommendations on the most suitable test methods for the high-temperature material characterization of the FRPs have been made.