Exergames have the potential to provide an accessible, remote approach for poststroke upper extremity (UE) rehabilitation. However, the use of exergames without any follow-up by a health professional could lead to compensatory movements during the exercises, inadequate choice of difficulty level, exercises not being completed, and lack of motivation to pursue exercise programs, thereby decreasing their benefits. Combining telerehabilitation with exergames could allow continuous adjustment of the exercises and monitoring of the participant’s completion and adherence. At present, there is limited evidence regarding the feasibility or efficacy of combining telerehabilitation and exergames for stroke rehabilitation.This study aims to (1) determine the preliminary efficacy of using telerehabilitation combined with exergames on UE motor recovery, function, quality of life, and motivation in participants with chronic stroke, compared with conventional therapy ; (2) examine the feasibility of using the technology with participants diagnosed with stroke at home; and (3) identify the obstacles and facilitators for its use by participants diagnosed with stroke and stroke therapists and understand the shared decision-making process.A mixed methods study protocol is proposed, including a randomized, blinded feasibility trial with an embedded multiple case study. The intervention consists of the provision of a remote rehabilitation program, during which participants will use the Jintronix exergame for UE training and the Reacts Application to conduct videoconferenced sessions with the therapists . We plan to recruit 52 participants diagnosed with stroke, randomly assigned to a control group  and an experimental group . The primary outcome is the Fugl-Meyer UE Assessment, a performance-based measure of UE impairment. The secondary outcomes are self-reported questionnaires and include the Motor Activity Log-28 , Stroke Impact Scale-16 , and Treatment Self-Regulation Questionnaire (motivation). Feasibility data include process, resources, management, and scientific outcomes. Qualitative data will be collected by interviews with both participants and therapists.At present, data collection was ongoing with one participant who had completed the exergame- telerehabilitation based intervention. We expect to collect preliminary efficacy data of this technology on the functional and motor recovery of the UE, following a stroke; collect feasibility data with users at home ; and identify the obstacles and facilitators for the technology use and understand the shared decision-making process.This paper describes the protocol underlying the study of a telerehabilitation-exergame technology to contribute to understanding its feasibility and preliminary efficacy for UE stroke rehabilitation.ClinicalTrials.gov NCT03759106; http://clinicaltrials.gov/show/NCT03759106.DERR1-10.2196/14629 P=.05) [In up to 85% of stroke survivors, sequelae persist in the upper extremity (UE) , resultiP=.05) ,8 and evexergames because the gaming goal is to enhance activity through various types of exercises. Among these exergames, some have been specifically designed for stroke rehabilitation, including for balance training as well as for training for both lower and upper limb deficits .intronix . Progresintronix . Howeverintronix ,6,9,10. To increase monitoring, a telerehabilitation system could allow such a follow-up through videoconferencing sessions between the user and the therapist. Such systems are increasingly used to provide remote rehabilitation services . Its effGiven the evidence for the efficacy of exergames as well as telerehabilitation for stroke rehabilitation, but the limited evidence of combining these technologies, such as in the VirTele program, the overall goals of this study are to explore its preliminary efficacy for UE rehabilitation and examine its feasibility for use with stroke survivors at home. More specifically, the objectives of this study are to (1) determine the preliminary efficacy of VirTele on UE motor recovery, function, quality of life, and motivation in participants with chronic stroke, compared with conventional therapy (GRASP); (2) examine the feasibility of using VirTele with participants diagnosed with chronic stroke at home; and (3) identify the obstacles and facilitators for the technology use by participants diagnosed with stroke and stroke therapists and understand the shared decision-making process.It is hypothesized that the exergame-telerehabilitation program will lead to greater UE motor recovery than usual care (GRASP) in participants with chronic stroke. We also hypothesize that the exergame-telerehabilitation program will have a greater impact on function, quality of life, and motivation in participants with chronic stroke.This is a mixed method study design consisting of a randomized, blinded feasibility trial with an embedded multiple case study that will take place in Montreal, Canada. The randomized feasibility trial is a two-arm, single-blind trial design in which eligible participants will be randomly allocated to an experimental (VirTele for 8 weeks) or control, usual care group (GRASP for 8 weeks). The feasibility trial captures both feasibility and preliminary efficacy outcomes. Publishing the feasibility results would provide a better understanding of the context in which the efficacy data were collected and a better interpretation of the final results ,14.Outcome measures will be assessed for both groups on four occasions: at baseline (T1), at the end of 2-month intervention (T2), after a 1-month follow-up period (T3), and after a 2-month follow-up period T4; .Randomization and evaluations will be performed by research assistants who are not involved in the study. All participants will provide informed written consent before enrollment. This study is registered at clinicaltrials.gov (NCT03759106) and has received ethics approval from the Research Ethics Boards of Centre for Interdisciplinary Research in Rehabilitation of Greater Montreal .At study entry, we will administer the Chedoke-McMaster arm component  to get eStroke survivors who have residual UE deficits and are no longer receiving rehabilitation services will be eligible for study participation if they fulfill the eligibility criteria. The inclusion criteria are as follows: first-time unilateral ischemic or hemorrhagic stroke or no residual deficits from a previous stroke and able to use the Jintronix system . The exclusion criteria are as follows: being medically unstable , severe cognitive or communication deficits, visual impairments limiting the ability to use the games, and UE mobility restrictions limiting the ability to play .We intend to recruit participants from the community and from the archives of rehabilitation centers (offline) situated in the Montreal, Sherbrooke, and Laval areas .Study therapists  from the different participating sites will also be included in the study to explore their experiences with the technology and comprehend the shared decision-making process underlying their choice of games and difficulty levels.To date, no studies have reported on the use of combining such technologies, such as in the VirTele program so that this randomized, blinded feasibility trial will provide evidence for the effect size. However, as the first estimate of effect size, a sample size of 52 participants has been calculated using the Fugl-Meyer Upper Extremity Assessment as a primary outcome and G*Power 3.1 . We assuFor the multiple case study component, four therapists and the first 10 participants from the experimental group will be invited to participate in interviews. However, the final sample size of the participants diagnosed with stroke will be adjusted depending on the qualitative data saturation .The participants in the experimental group will receive the VirTele program. VirTele is an 8-week home program during which participants will use the Jintronix exergame  for UE tThe Reacts application is an interactive audio-video platform that allows secure communication between therapists and participants using standard computer or tablet technologies . The appReacts can also be used alone, making it possible for the participant to interact with the therapist when finishing the exercises. These sessions provide an opportunity to discuss difficulties regarding playing games and to subsequently modify the rehabilitation program according to the participant’s abilities and preferences. This shared decision-making process can help foster motivation to make health-related decisions  and specially to continue rehabilitation after the end of the intervention. This shared decision-making process will aim to increase the empowerment of the participant, facilitating subsequently the transfer of the functional gains acquired in the context of the study into real life. This transfer occurs by identifying ways of reproducing the tasks of the game in activities of the daily life and increasing the use of the affected UE in the long term. Participant empowerment will be further encouraged by using motivational interviewing based on self-determination theory (SDT) principles during telerehabilitation sessions ,22. ThisThe Jintronix exergame consists of six UE games played for varying amounts of time and at different difficulty levels , which can be tracked remotely by the therapist asynchronously through the data provided in Jintronix Web-based portal. The therapist is also able to modify the difficulty parameters according to the performance data recorded on the portal. This exergame uses a Kinect camera, which captures the person’s body movements without wearing sensors. Participants will be invited to use Jintronix at least five times a week for 8 weeks, for 30-min sessions, performing a total of 20 hours of exercise. A minimum of 15 hours is suggested for an intervention to result in a moderate improvement in activities related to daily living following a stroke .Jintronix includes an automated log system, which records the active time spent by participants in each game and the score achieved. The therapist can access the exergame interface at any time to monitor the participant’s progress and adherence to the exercise program and modify the difficulty level.Before starting the intervention, therapists will receive training in motivational interviewing  to ensurAll participants in the experimental group will participate in a 30-min training session with the technician responsible for the installation of the technology to learn to use the VirTele technology.The participants in the control group will receive GRASP based on their UE function. It includes strengthening exercises of arm and hand, range of motion, and functional arm activities . The proAt study entry, the participant’s sociodemographic information  will be collected for descriptive purposes.For the randomized, blinded feasibility trial, several outcomes measures will be used to address the different objectives. The first objective of the trial is to determine the preliminary efficacy of VirTele on UE motor recovery, function, quality of life, and motivation, in participants with chronic stroke, compared with conventional therapy (GRASP).The Fugl-Meyer Upper Extremity Assessment will be the primary outcome to determine the efficacy of the technology for UE motor control recovery. This is a performance-based measure of UE impairment ,26. It iThe secondary outcomes are self-reported questionnaires and include the Motor Activity log 28 ,28, the The Motor Activity log 28 is a self-reported measure of UE use ,28. ThisThe second objective of the randomized, blinded feasibility trial is to assess the feasibility of using VirTele with participants diagnosed with stroke at home. Feasibility data include indicators of process, resources, management, and scientific feasibility . In TablA trained assessor who is not involved in the delivery of interventions and blinded to group assignment will be responsible for the face-to-face administration of the outcome measures.Individual semistructured interviews of 30 min will be conducted with the first 10 participants from the experimental group (n=10) after the end of 8-week intervention. The interview guide will be developed based on the Unified Theory of Acceptance and Use of Technology (UTAUT) conceptual framework . This thThe study therapists will also be interviewed to describe their experience with VirTele (obstacles and facilitators), explore their behavior to see if they align with SDT, and investigate the shared decision-making process underlying their choice of games and difficulty levels. Individual semistructured interviews of 30 min will be used. The UTAUT and the SDT will serve as basis for the interview guide development.All interviews will be voice recorded and transcribed verbatim. A logbook and reflective notes will also be taken.Statistical analysis of the quantitative data will be performed using the Statistica software. Descriptive statistics will be used to report sociodemographic characteristics of participants  in both groups . To address the first objective of the randomized, blinded feasibility trial, a mixed model approach will be applied for primary and secondary outcomes measure. Each model will contain one between-subject effect factor , one within-subject effect factor , and two covariates (gender and age) factors, which may impact exergame use. For each outcome measures, residual plots will be examined to verify normality and identify the best covariance structure. All the outcomes measure changes will be compared with the minimal clinically important differences. The effect size of the comparison between experimental and control groups will be calculated to estimate the final sample size. To address the second objective of the randomized, blinded feasibility trial, descriptive statistics  will be used to highlight the amount of exercise performed, the occurrence of adverse events, and participant and therapist satisfaction level in the experimental and control groups.For the qualitative data, thematic analysis will be conducted for each case group. Transcripts will be coded using a predetermined coding scheme  based onWe expect to (1) collect preliminary efficacy data of this technology on the functional and motor recovery of the UE, following a stroke; (2) collect feasibility data with users at home ; and (3) identify the obstacles and facilitators for the technology use and understand the shared decision-making process during the VirTele program.At the time of this manuscript submission, data collection was ongoing with one participant who had completed the study  and used VirTele for 2 months  Multime. At thisThis paper describes the research protocol for a mixed method study, including a randomized, blinded feasibility trial with an embedded multiple case study. The aims of this study are to (1) determine the preliminary efficacy of VirTele on UE motor recovery, function, quality of life, and motivation in participants with chronic stroke, compared with conventional therapy (GRASP); (2) examine the feasibility of using VirTele with participants diagnosed with chronic stroke at home; and (3) identify the obstacles and facilitators for the technology use by participants diagnosed with stroke and stroke therapists and understand the shared decision-making process.We have chosen a mixed study design because the use of both qualitative and quantitative approaches makes it possible to construct a more complete image of the studied phenomenon. The combination of the two methodologies should be approached not from the point of view of their differences but from the complementarities they can bring to the study . If the Participants will be recruited from different sites to increase the representativeness of the target population in the region. The control group will not receive any motivational interviewing or follow-up to keep the standard aspect of therapy that corresponds most to the clinical reality. This will enable us to identify the added value of the VirTele program compared with GRASP. After outcome measures collection in T1 and T2, to compare the effect of each intervention within and between groups, we will collect additional measures at T3 and T4 to evaluate the retention of gains.The multiple case study provides an in-depth description of stakeholders use experience and potential use of VirTele program. In this study, we will have two case groups: participants diagnosed with stroke and study therapists . Each of these cases experiences the intervention differently, and it is therefore essential to report them through interviews. The variation of the cases makes it possible to increase the variation of the experiences and thus to increase the robustness of the qualitative results . UTAUT aThe results of this study will allow to verify if all elements of the protocol work well together to conduct a broader future study. We have tried to provide as much detail as possible about the various processes and steps of the protocol to facilitate its reproduction by other studies that seek to develop tools for the remote management of chronic diseases.This project will also provide preliminary evidence of the efficacy of VirTele on motor and functional recovery of the UE following chronic stroke for future guidelines review, although studies caution us about solely using results from feasibility studies to establish intervention efficacy .Extending rehabilitation following a stroke with remote services may be a promising strategy to overcome the limited resources in the health system. The VirTele program is a new approach that may provide stroke survivors continuous and remote access to rehabilitation services. This paper describes the protocol underlying the study of this technology to better understand how it can be used among different stakeholders and explore its preliminary efficacy in a chronic stroke population.


It can be challenging to detect tumor margins during surgery for complete resection. The purpose of this work is to develop a novel learning method that learns the difference between the tumor and benign tissue adaptively for cancer detection on hyperspectral images in an animal model. Specifically, an auto-encoder network is trained based on the wavelength bands on hyperspectral images to extract the deep information to create a pixel-wise prediction of cancerous and benign pixel. According to the output hypothesis of each pixel, the misclassified pixels would be reclassified in the right prediction direction based on their adaptive weights. The auto-encoder network is again trained based on these updated pixels. The learner can adaptively improve the ability to identify the cancer and benign tissue by focusing on the misclassified pixels, and thus can improve the detection performance. The adaptive deep learning method highlighting the tumor region proved to be accurate in detecting the tumor boundary on hyperspectral images and achieved a sensitivity of 92.32% and a specificity of 91.31% in our animal experiments. This adaptive learning method on hyperspectral imaging has the potential to provide a noninvasive tool for tumor detection, especially, for the tumor whose margin is indistinct and irregular. Orophary cancer is a common cancer worldwide and in recent years its incidence increased in a fast pace in both America and Europe . More thHyperspectral imaging (HSI) is a technology that can acquire a series of images in many adjacent narrow spectral bands and reconstruct the reflectance spectrum for every pixel of the image . By measHyperspectral images, known as hypercubes, contain rich information on a wide range of spectra with a high spectral resolution , hence, However, these technologies require domain-specific knowledge to extract discriminant data to convert suitable features. In contrast to these conventional machine learning techniques, deep learning models can learn representations of data with multiple levels of abstraction, thereby can discover intricate structures in high-dimensional data with very little engineering by hand . ConvoluSince hyperspectral imagery has the system noise and image artifacts arising from uneven surface illumination, the tumor margin is irregular and unclear. So it is difficult to distinguish a tumor from surrounding normal tissue. In this study, we proposed an automated cancer detection algorithm for highlighting the tumor by adaptive auto-encoder network learning. Auto-encoder is an unsupervised deep neural network that can learn the inherent features and extract the suitable representation from complex data automatically. We involved the auto-encoder network to learn and recognize the depth features of pixels in hyperspectral imagery for the initial cancer detection. Each pixel is assigned a weight according to its classification result. The proposed adaptive auto-encoder learning method is performed on these weighted pixels and is trained to correct the misclassified pixels for the improvement of the detection performance. In this study, we demonstrate the efficiency and effectiveness of the auto-encoder and adaptive deep learning in HSI for head and neck cancer detection in an animal model. The method and experiments are described in the following sections.Hyperspectral images were obtained by a wavelength-scanning CRI Maestro in vivo imaging system. This instrument mainly consists of a flexible fiber-optic lighting system, a solid-state liquid crystal filter, a spectrally optimized lens, and a 16-bit high-resolution charge-coupled device. For image acquisition, the wavelength setting can be defined within the range of 450 to 950 nm with 2-nm increments. Further details can be referred in our previous paper , 55.The proposed adaptive deep learning method for cancer detection on HSI contains four parts: pre-processing, deep feature learning, adaptive weight learning, and post-processing. Figure I, is calculated byIraw is the raw image, Iwhite is the white reference image (100% reflectance) obtained by placing a standard white reference board in the field of view, and Idark is the dark image (0% reflectance) was acquired by keeping the camera shutter closed. These reference images were used to calibrate hyperspectral raw data before image analysis.The acquired hyperspectral images were saved in a raw format, and correction was made with a white and a dark reference to remove the influence of the dark current and obtain a relative reflectance image. The corrected image, k inputs, one hidden layer of d units, one reconstruction layer of k units, and an activation function. We illustrate the architectures of an auto-encoder in Fig. An auto-encoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner . It has X ∈ Rk to the hidden layer and produce the latent activity Y ∈ Rd. The decoder is to map Y to an output Z ∈ Rk, having the same number of nodes as the input layer, and with the purpose of reconstructing its own inputs X. We can get the Y and Z byw1 and w2 are the weight of input-to-hidden and the hidden-to-output, and b1 and b2 are their bias, f(p) is the activation function. In our method, it is a sigmoid function like:An auto-encoder consists of two parts, the encoder and the decoder. The encoder is to map the input Based on the structure of the auto-encoder, we use the spectral features of each pixel as the input, and train the network by iteratively updating the weights and biases to minimize the loss function of reconstruction error. To improve the robustness of the auto-encoder, in addition to the mean squared error between the input features and the reconstructed features, the L2 regularization and the Kullback-Leibler divergence based sparsity regularization is incorporated into the loss function. The learned feature that lies in the hidden layer is a learned deeper feature. The decoder part is been removed and the softmax layer is added into the network for the classification of cancer and normal tissue. The framework is shown in the Fig. The auto-encoder network can identify the cancer pixels and healthy pixels. Then we can obtain the initial detection result of cancer and the output hypothesis of pixels.mask is the initial cancer detection, Pij is the i-th and j-th pixel in the mask and weight (Pij) is the weight of Pij. The weight_se(Pij) means to focus on the weights of the false negative pixels and the weight_sp(Pij) means to focus on the weights of the false positive pixels. The se (mask) is the sensitivity of the initial cancer detection and the sp (mask) is the specificity of the initial cancer detection. The Pri (Pij) is the prediction of the initial model for the pixel Pij, true (Pij) is the true label of the pixel Pij, 0 means the healthy tissue and 1 means the cancer tissue.Because of the characteristics of medical images, the tumor margin is irregular and indistinct. The values of pixels in tumor are similar with those of pixels in healthy tissue. So, it is difficult to distinguish the tumor from its surrounding normal tissue for any learning method. Hence, we highlight the tumor by adjusting its weight adaptively in two aspects according to the misclassified pixels. The misclassified pixels can be divided into two types, the health pixels which are misclassified into cancer  and the cancer pixels which are misclassified into health . The adaptive weight is assigned to each pixel according to its output hypothesis from the initial model, and it can be calculated byhypercube_update is the updated hypercube and hypercube is the original hypercube, hypercube_update(Pij) means the value of Pij in the updated hypercube and it is the product of the value of Pij in the original one and the weight of Pij. The main aim of updating the original hypercube is to highlight the tumor most conducive to learning adaptively. Based on the updated hypercube, the deep features can be retrained, and an adaptive auto-encoder network model can be obtained.Eq.  is the dweight_se could assign the weights to those false negative pixels and the updated image could focus on those pixels. So the new auto-encoder learner on the updated images tends to expand the relevant pixels to improve the sensitivity of tumor detection. If the detected tumor contained some irrelevant regions, then weight_sp could assign the weights to those false positive pixels, and the updated image could highlight those pixels. So the new learner tends to eliminate the irrelevant pixels to improve the specificity of tumor detection. Thus, the pixels classified correctly keep their correct prediction while the pixels misclassified change their values adaptively. Therefore, the adaptive learner heightens the ability to identify the tumor and healthy tissue.Eqs.  and 5)  5) show In this paper, the refined type improving the sensitivity or specificity will be determined by experiments according to the detection performance of training images.Since our method is based on the classification of each pixel, the detected tumor may contain some noise and holes. The flood-fill operation is used to fill holes in the segmented binary image and the biggest connected component is chosen as our detected cancer tissue.All methods were carried out in accordance with the approved Institutional Animal Care and Use Committee protocol (YER-2003103-042918BN) and the relevant guidelines and regulations of Emory University. We acquired the hyperspectral reflectance images from 12 tumor-bearing mice approximately 2 weeks post-tumor cell injection. The reflectance images contained 251 spectral bands and the image size on each spectral band was 390 × 435. Therefore, the data cube collected was a three-dimensional array of the size 390 × 435 × 251. In this study, tumor cells had green fluorescence protein (GFP) signals and thus GFP images were also acquired as the reference standard to evaluate the proposed tumor detection algorithm.We conducted leave-one-out cross-validation experiments for the tumor detection in hyperspectral images. We take each hyperspectral image as the testing sample in turn, and the 11 remaining samples as the training samples.The performance of auto-encoder could be affected by its reduced dimensionality. We test the dimension of compressed features from 20 to 60, step 5 on the training set. For each mouse, we train and test on its training set, and calculate the detection accuracy of each training samples, and obtain the average detection accuracy. The results of 12 mice named from #1 to #12 are shown in Fig. Next, we recorded the accuracy obtained by the adaptive sensitivity-weighted learning and specificity-weighted learning on the training set. The initial detected tumor of each mouse in the training set is refined by adjusting its sensitivity-weight and specificity-weight. The average accuracy is shown in Fig. To obtain the effective information, the auto-encoder is adopted to learn to compress the 251 wavelengths from 450 to 950 with 2 nm increments into a short feature by extracting the useful characters and ignoring noise. We randomly choose one mouse and use the auto-encoder to obtain the compressed features. Its average spectral values for the cancerous tissue and the healthy tissue of 251 wavelengths and 60 compassed features are shown in Fig. In addition, we compare the auto-encoder with the CNN-based semantic segmentation neural network, such as FCN , SegNet To see the advantage of adaptive weight in the detection of cancer tissue, we compare the detection performance of the original learning method and the adaptive weight learning method. The compared results are shown in Fig. We show the qualitative results of cancer detection on five mice in Fig. In Fig. Table In this study, we proposed automated detection method for head and neck cancer using the adaptive deep learning on hyperspectral imagery in an animal model. Auto-encoder network model is involved to extract the deep features from a hyperspectral imagery with size of 390 × 435 × 251 and distinguish the cancerous tissue from its surrounding normal tissue. Because of the noise and uneven surface and so on, the detected cancer region is not satisfactory. To improve the initial performance and obtain a complete tumor, the adaptive auto-encoder network model is proposed, which focus on the misclassified pixels and enhance to learn for the misclassified pixels. The method is shown to classify the tumor region with high sensitivity, specificity, and accuracy.Since reflectance hyperspectral images contain 251 spectral bands from 450 to 950 nm with 2 nm increments, and each hyperspectral image contains over millions of reflectance spectral signatures, it is difficult to extract discriminant features from the huge data by hand. Deep learning methods can learn features by building high-level features from low-level ones and automatically discover the features needed for cancer detection. The auto-encoder is an unsupervised deep neural network that tries to denoise the inputs automatically by finding the latent representation from which to reconstruct the original input, hence it is especially suitable for describing the hyperspectral data. As shown in the Fig. Although auto-encoder can extract the useful information for detecting the cancer tissue, the blood vessel or uneven surface make it difficult to identify the complete tumor from the normal tissue. Since blood vessel or uneven surface could misrepresent the tissue structures, neither the traditional classification methods nor the deep learning methods could detect the tumor with high accuracy by learning the intensity or the distribution of intensity. The proposed adaptive auto-encoder network can predict the misrepresented tissue structures into their true classes by adaptively weighting those misclassified pixels, and thus greatly improve the performance of tumor detection, as shown in Figs. The automatic detection algorithm was written and run in MATLAB on Intel Core 2.60GHz CPU with 16GB of RAM. The time for normalization, deep feature extraction, cancer detection, post-processing is about 0.1 s, 2.8 s, 3.2 s, and 0.02 s, respectively. The total running time is about 6 s for per hyperspectral image. It greatly improved the efficiency of cancer detection compared with the method  using 45In this study, an adaptive deep learning framework was proposed and validated for head and neck cancer detection using HSI in an animal model. This algorithm extracted the deep feature of hyperspectral images for reducing the dimensionality effectively and better characterizing the cancerous tissue. The adaptive weight learning could improve the cancer detection performance by focusing on harder-to-classify pixels. In the head and neck cancer mouse model, the proposed cancer detection method was able to obtain a high sensitivity and specificity. The results demonstrated that the HSI combined with deep learning technique may enable accurate and fast detection of cancers in a noninvasive manner and may provide a promising tool for future clinical applications.


The treatment of choice for advanced non–small cell lung cancer is selected according to the presence of specific alterations. Patients should undergo molecular testing for relevant modifications and the mutational status of EGFR and translocation of ALK and ROS1 are commonly tested to offer the best intervention. In addition, the tests costs should also be taken in consideration. Therefore, this work was performed in order to evaluate the cost-effectiveness of a unique exam using NGS (next generation sequencing) versus other routinely used tests which involve RT-PCR and FISH.The target population was NSCLC, adenocarcinoma, and candidates to first-line therapy. Two strategies were undertaken, strategy 1 corresponded to sequential tests with EGFR RT-PCR, then FISH for ALK and ROS1. Strategy 2 differed from 1 in that ALK and ROS1 translocation testing were performed simultaneously by FISH. Strategy 3 considered single test next-generation sequencing, a platform that includes EGFR, ALK and ROS1 genes. A decision tree analysis was used to model genetic testing options. From the test results, a microsimulation model was nested to estimate survival outcomes and costs of therapeutic options.The use of NGS added 24% extra true cases as well as extra costs attributed to the molecular testing. The ICER comparing NGS with sequential tests was US$ 3479.11/correct case detected. The NGS improved a slight gain in life years and QALYs.Our results indicated that, although precise, the molecular diagnosis by NGS of patients with advanced stage NSCLC adenocarcinoma histology was not cost-effective in terms of quality-adjusted life years from the perspective of the Brazilian supplementary health system. Advanced lung cancer has played a key role in the development of medicines aimed for individualized therapy. Currently, it is recommended that all patients who are candidates for chemotherapy treatment should undergo molecular testing to determine the best treatment clinically available –3.Kirsten rat sarcoma viral oncogene homolog) and fusion of EML4-ALK1 (echinoderm microtubule-associated protein-like 4 – anaplastic lymphoma kinase), along with other ongoing clinical trials, such as MET, BRAF, RET (Echinoderm Microtubule Associated Protein like 4-AL-Kinase 1) and ROS1 (receptor tyrosine kinase 1) [: EGFR [The Epidermal Growth Factor Receptor (EGFR) was the first antigenic target used as a guideline for targeted lung cancer therapy, followed by other targeting markers, such as KRAS  and immunohistochemistry (IHC), among others, were developed and are being used for the clinical evaluation of oncogenic markers. Due to technical limitations and the small amount of material obtained from biopsies, none of these techniques can be scaled to meet the increasing number and variety of genomic changes. This has led to the development of parallel multi-genic DNA sequencing platforms such as next-generation sequencing (NGS), that allows for the simultaneous analysis of hundreds of genetic alterations in a single test .The major impediment for effective implementation of individualized therapy is the access to companion tests and drugs, due to the high costs that health systems generally cannot afford. Brazil is a middle-income country that had an estimated 208 million inhabitants and where 31,270 new lung cancer cases were expected in 2018 . Within The NGS technique has been raising expectations about the possibility of performing a single test to define prognosis and treatment. However, this technique is associated with high costs, raising doubts about whether its dissemination is cost-effectiveness for health systems. Few countries evaluated the varying strategies for the use of molecular tests , 18. To A cost-effectiveness study was done in reference to the population of adult patients affected with adenocarcinoma of NSCLC stage IV. The economic analysis considered the long-term effects of technical performance and the accuracy of the following molecular testing strategies used for tumor tissue biopsies:Strategy 1: RT-PCR for mutation identification of the EGFR gene. If negative, the individual is sent for ALK gene fusion testing; should this test also be negative, the patient continues on to ROS1 gene testing. Strategy 2: differs from strategy 1 in that ALK and ROS1 translocation testing are performed simultaneously by FISH. Strategy 3: considers new intervention along with next-generation sequencing, with a platform that includes EGFR, ALK and ROS1 genes in a single test.A decision tree model was built comparing the three strategies based on prevalence of genetic alterations, accuracy, and tests performance Fig. .Fig. 1DThe transition state models were constructed based on different available information. Positive results for EGFR led to the use of gefitinib in the first line , 20 follRelevant outcomes estimated in the study were the costs associated with molecular diagnosis and treatment, correctly diagnosed cases (true positive and true negative) from the different diagnostic strategies, years of life gained and quality-adjusted life year (QALY). The time horizon is 5 years and all costs and health outcomes were discounted at an annually rate of 5% according to Brazilian health economic evaluation guidelines .The parameters utilized in the model, as well as the value limits used in the sensitivity analysis, are described in Table The incremental cost-effectiveness ratio (ICER) was calculated using the ratio of the differences between costs and QALYs among strategies. The ICER represents the incremental cost of a strategy for gaining a single unit of health benefit. To assess the degree of uncertainty of the results, the probabilistic sensitivity analysis was performed by varying all parameters within the uncertainty interval, according to their distribution Table .A decision tree model was constructed comparing the three strategies (described in the methods section) based on prevalence of genetic alterations, accuracy, and performance of the tests Fig. . Our anaFrom the tree model results, a microsimulation model was designed to estimate the effectiveness of the diagnostics up through treatment, based on free survival data, progression, and overall survival rates of the different therapies. The study modelled the complete course of treatment over the entire life expectancy of the patients through a microsimulation model, which included second and third treatment lines. The clinical practices regarding the treatment lines used were selected from the therapeutic guidelines and validated based on consultation with specialists working in the Brazilian private health insurance sector to reflect the practice in the country.Three transition state models were constructed, corresponding to disease progression and treatment until the third line Fig. .Fig. 2SThe use of NGS identified an additional benefit of 24% of correctly diagnosed cases at an incremental cost of $800.76. The ICER comparing NGS with sequential tests was US$ 3479.11 for each correct case detected. The comparison of strategies 2 and 1 (2:1), indicated that the ICER was US$ 961.46 for each correct case detected .The first part of the decision analysis model considered only the intermediate effects, which are the test results. To identify if, apart from the test results, the choice of diagnostic strategy has an impact on survival, a transition state model was used to consider the effectiveness of treatment. Regarding survival, the difference in incremental years of life gained, and QALYs between the strategies was very small. The cost-effectiveness plan in Fig. Cost-effectiveness acceptability curves were constructed to show the probability of each strategy being considered as cost-effective in relation to another, using values that health plans could afford to pay per QALY Fig. . Our resThis study represents the first economic evaluation analyzing NGS and the cost-effectiveness ratio from the perspective of the private health insurance sector in Brazil. Economic assessments focusing on the comparison of NGS use with other tests for gene changes present in lung cancer are still relatively scarce in the literature  with mosOne of the outcome measures defined for this study considered the number of cases correctly identified by each of the diagnostic strategies. The results showed that the single test with NGS had a greater likelihood of producing a correct diagnosis (summation of true positive and negative cases) — 96.3% — than the diagnostic routine based on individual tests for EGFR, ALK and ROS1 (68%) and the test for EGFR, followed by simultaneous testing for ALK and ROS1 (72.6%). To this end, the high sensitivity, and specificity of the NGS, in both cases exceeding 99%, outperformed sensitivity compared to the EGFR test used (97.5%), however with less specificity (86.7%). The sensitivity and specificity of the FISH test for ALK and ROS1 did not compete for differences in the number of cases correctly diagnosed, since they were considered in the case of reference as being 100%. In this sense, although the number of false-negative patients for EGFR is small (depending on the sensitivity of the test) the smaller specificity leads to a greater number of false-positive results for EGFR, which, in practice, results in the loss of detection of possible cases with ALK or ROS1 mutations.Some reports indicated that NGS surpasses Sanger sequencing in terms of sensitivity. For example, using 80 small routine samples from routine biopsies and cytology, De Biase compared NGS with Sanger sequencing, and demonstrated that NGS improved detection of EGFR mutations, particularly in samples with low tumor cell content . In the The cost-effectiveness study modelled the insertion of the recommended therapies and estimated long-term consequences of the different test-treatment strategies. The objective was to establish a link between the diagnostic performance, its clinical value , and the incremental cost-effectiveness ratio. Thus, when the treatments and their results were incorporated into the model, the effectiveness of the tests were diluted between arms. The NGS was more effective than the other two diagnostic strategies, with small gains in years of life and quality adjusted life years . It was also more costly , resulting in a high ICER, about US $ 214,000.00 per QALY gained. The small gain in terms of years of life and QALYs can be explained by small differences in the overall survival between treatments. Although most TKI-related clinical trials show statistically significant differences in tumor response rate (ORR) and progression free survival (PFS), they are not able to demonstrate significant differences in overall survival (OS) between these and standard chemotherapy .The lack of association between the differences observed in the ORR and PFS, and the result in terms of OS is attributed, among others, to crossover between arms of clinical trials, with migration of patients from the control groups to the intervention arm, in the presence of certain predefined events such as disease progression. If survival is extended to patients who migrate between the randomized arms outcomes, such as OS, the differences could be reduced in such a way that there is no longer any statistical significance. The intention-to-treat analyses may underestimate the advantages in terms of OS and the economic efficiency of the new intervention .The differences in testing costs are much more significant and impactful, and even the lower limit of the price range for the NGS practiced in the Brazilian market (US$ 1502.51) far exceeds the upper limits of the bands raised for the other two tests . New technologies tend to be introduced in the market at higher values, with their prices tending to decline as they become more widespread. This may also occur with the NGS, with future price reductions below the lower limit currently used, which could make the strategy based on this test more favourable.Another factor that increases the advantage of NGS over other tests is the ability to generate more information in a single test . A Brazibias resulting from the patients living longer and adding costs to the model in relation to treatment, since in the first line with target therapy patients use the drug daily until progression. Thus, the individuals who contribute mostly to the strategy, in terms of survival, also contribute for the higher costs. Importantly, the construction of cost-effectiveness acceptability curves allows for summarization of the uncertainties about the estimates made. In this case, the acceptability curve makes it clear that the probability of the NGS test being more cost-effective than the other diagnostic-treatment strategies is quite small: less than 40% for values that exceed US$ 214,000.00 per QALY.Another important aspect regarding treatment costs refers to the Doble et al. 2017, focusing on the fourth-line treatment of metastatic lung adenocarcinoma, compared three diagnostic-treatment strategies: (i) use of NGS and treatment with target therapies only in patients with detected changes, with the remaining receiving chemotherapy (vinorelbine) or supportive care; (ii) no additional tests, with chemotherapy, and (iii) no additional test with supportive treatment . A decis. This waiting time was not included in the present study. It should be noted that although NGS is a complex test [Some important aspects related to the tests were not incorporated into our model. An important assumption underlying this model is that the tests lead to different treatments, ignoring other factors that could contribute to variations in the outcome of therapy such as the time for the availability of test results, relevant to initiating therapy. Some international guidelines recommend EGFR and ALK results to be available within a maximum of 2 weeks (10 working days), from receipt of the specimen in laboratory . This walex test , its clilex test . Howeverlex test . In addiFor all the diagnostic strategies examined, the probability of not obtaining the result was considered, either due to insufficient tumour cells or due to poor sample quality. The simulations considering 1000 tests showed that NGS would result in 34 inconclusive tests, a number well below strategy 1, where testing would result in 130 inconclusive cases for EGFR mutation and 55 cases with FISH. The better performance of the NGS over other tests could be explained by its ability to detect changes in lower sample concentrations. Le Mercier (2015)  showed tIn the present study, in all cases of unavailable outcome, the individual followed directly to treatment with conventional chemotherapy for the first line, not being defined a second tumour biopsy and molecular retesting since this was considered as not being the most frequent pattern of care in Brazil and also because the material obtained in the repeated biopsy could also be insufficient or inconclusive. Repeated biopsy and retesting could result in additional costs for the three screening strategies. Given the large cost-per-test differences, it is unlikely that this would change the ICER to levels that could be considered acceptable. An internationally agreed standard for sequencing analysis and data interpretation for clinical, public health, and regulatory purposes must be developed to make NGS more reliable . The useThe effort to use prevalence values for the various genetic alterations from national studies, and the choice to make a microsimulation model to design the gains in survival between the different strategies, are important strengths of our analysis. Although the number of studies in our country is quite small, the significant regional differences indicated justify efforts to search for and use measurements based on the Brazilian population. Likewise, the attempt to obtain data closer to that practiced in the market for both comparative tests and from a set of commercial companion tests. We also sought to aggregate drug costs used in pre-chemotherapy and room rates according to the infusion time of each chemotherapy protocol. The microsimulation model was a good choice to simulate a cohort of individuals, from their respective attributes, according to the survival curves present in the literature. In other models, such as Markov transition state models, it is possible to aggregate survival data, nevertheless assumptions must be made such as grouping individuals in the same health state. Thus, the costs and outcomes are calculated from each state, which may result in losing information at the individual level.The NGS has been finding barriers to its coverage not only in precision therapy, but also in screening for hereditary cancer, among other indications. Although the benefits of a larger diagnosis are recognized, this diagnostic model contrasts with the usual single test/single result payment model in health systems. For the payer, generating information beyond what is needed immediately in the assistance may be a source of uncertainties as the economic impact that these changes may produce.The data presented indicated that, although the molecular diagnosis by NGS of patients with NSCLC with advanced stage adenocarcinoma histology allows for a greater number of correctly diagnosed gene alterations cases, this test was not cost-effective in terms of quality-adjusted life year from the perspective of the Brazilian supplementary health system. NGS was not deemed cost-effective compared with individual directed genomic tests for EGFR, ALK, and ROS1 in a model reflective of the current therapeutic limitations available in Brazil. As new therapies for these genomic alterations and others become available, the value of NGS testing should be reconsidered. It is possible that in the near future, the NGS may aggregate more diagnostic information that positively impacts treatments’ effectiveness in addition to optimizing the use of other types of biopsy. Considering this increase in effectiveness, and the possibility of reducing the cost of testing through its expanded use, in a short time the results of the NGS can become cost-effective or even cost-saving.This study is part of an effort to integrate discussions related to the companion tests for detection of gene alterations and target therapy in the Brazilian health system. Nevertheless, other studies should be performed integrating liquid biopsies and a panel of comprehensive mutations related to rarer genetic alterations that possess useful target drugs.


Nowadays, the Global Navigation Satellite Systems (GNSS) technology is not the primary means of navigation for civil aviation and Air Traffic Control, but its role is increasing. Consequently, the vulnerabilities of GNSSs to Radio Frequency Interference, including the dangerous intentional sources of interference , raise concerns and special attention also in the aviation field. This panorama urges for figuring out effective solutions able to cope with GNSS interference and preserve safety of operations. In the frame of a Single European Sky Air traffic management Research (SESAR) Exploratory Research initiative, a novel, effective, and affordable concept of GNSS interference management for civil aviation has been developed. This new interference management concept is able to raise early warnings to the on-board navigation system about the detection of interfering signals and their classification, and then to estimate the Direction of Arrival (DoA) of the source of interference allowing the adoption of appropriate countermeasures against the individuated source. This paper describes the interference management concept and presents the on-field tests which allowed for assessing the reached level of performance and confirmed the applicability of this approach to the aviation applications. Civil aviation and Air Traffic Control (ATC) are deeply tied to localization and navigation systems. Such systems are based on several technologies installed either on board or on the ground, including radio-beacons, RADAR, magnetic compasses, inertial navigation systems, and satellite positioning systems . Global The International Civil Aviation Organization (ICAO) and the European Organisation for Civil Aviation Equipment (EUROCAE) are working on shaping these trends, promoting the discussion about the evolution of the role of GNSS in aviation, while in parallel fostering the necessary technological advancement ,4. For eOn the other hand, the well-known vulnerabilities of GNSSs to Radio Frequency Interference (RFI) raise concerns and special attention also in the aviation field . Facts wThis panorama, which has been alerted among others by International Air Transport Association (IATA) , urges fThe objective of this paper is to present the results of an on-field test campaign of a novel, effective, and affordable concept of GNSS interference management for civil aviation, developed under a SESAR ER initiative . This nepre-correlation block, the received radio frequency signal from each antenna is pre-processed before entering the receiver operations, in order to detect and classify a possible jamming signal and to estimate its DoA. In the post-correlation block, the receivers’ outputs in the form of code and carrier pseudorange measurements are used to detect possible counterfeit (spoofed) signals in the ensemble processed by the receivers and to determine their DoA. On the ground, a hybridization server implements the collaborative interference management: it receives measurements from all the aircraft in the area regarding the presence of interference  and combines the cooperative information through a hybridization mechanism, e.g., based on machine learning or particle filtering approaches. A schematic block diagram of such a collaborative approach is depicted in The novelty of the proposed concept is two-fold, both on board and on the ground. On board, the signal processing architecture is developed as an external two-blocks add-on of existing GNSS receivers, as sketched in With respect to , where tThe on-field tests, together with the practical implementation of the interference detection and direction finding algorithms which takes into account the existing or realistically expected aircraft equipment, are the main contributions of this paper with respect to existing sources in literature. Indeed, the context of active GNSS interference management from on-board aircraft reusing omnidirectional navigation antennas is new by itself in the civil aviation field. To the best of the authors’ knowledge, no literature exists that addresses in an integrated way the various interference types encountered in GNSS and that explicitly deals with the stages to counteract this interference Detection and Collaborative Localization (D&CL)The integrated GNSS interference management aims to provide an accurate position of the interference source sensed on-board the aircraft and to report the information to ATC. Depending on the complexity of the system, two modes of operation have been defined to provide accurate localization:The concept of autonomy here is based on the idea that the aircraft relies only on the data recorded on-board to localize the interference source. During nominal operation (detection), the aircraft is continuously monitoring the presence of jamming or spoofing interference, using specific algorithms to detect each type of attack. When an interference is detected, the aircraft automatically starts the localization. At every epoch, the aircraft estimates the DoA of the interfering signal, using the corresponding algorithm for each type of interference. In addition, the aircraft integrates the localization computed at each epoch along the trajectory where the interference is affecting. It provides an accurate localization of the interference thanks to benefit from the movement of the aircraft with respect to the interference source. As soon as the aircraft has estimated a ‘reliable’ position of the interference source, the ATC has to be reported with an alert and some additional information. The information reported by the aircraft includes the type of interference, the estimated position of the interference source, a time-tag, the error in the estimated position and the estimated affected volume (radius of the affected area at certain flight level). For the sake of automation and with regard to the low data required to transmit the information from the aircraft to the ATC, it is recommended to use a data link .This mode is collaborative in the sense that the localization estimated by multiple affected aircraft is integrated on-ground, potentially achieving a more accurate localization of the source. During nominal operation (detection), each aircraft is continuously monitoring the presence of interference, as in the previous mode D&AL. The main difference of D&CL mode with respect to mode D&AL is the need of a ground infrastructure. The estimated DoAs of the interfering signal and associated estimation error statistics computed by each aircraft at every epoch are transmitted to the ground infrastructure, which can compute a better localization of the source thanks to the multiple sources of information .It is interesting to notice that, in D&CL mode, aircraft transmit the information to the ground infrastructure, whereas in D&AL mode it is transmitted to ATC. Nevertheless, the same information is transmitted in both modes and therefore the same data-link can be used.GNSS antennas layout: Currently, two GNSS omnidirectional antennas are normally available for navigation and placed on top of the fuselage. Moreover, an additional third omnidirectional antenna is required for interference localization. Layout of a right triangle with baselines between 1 and 3 m is a suitable configuration for the GNSS antennas to support detection and localization of both jamming and spoofing.Data processing: Additional hardware and software is needed on-board to process the signal from the GNSS antennas and to implement the detection and localization algorithms. One of the most important constraints assumed in this work is to minimize the installation of additional equipment on-board the aircraft. Taking this into account, the elements that require modification are:x and y reference axes;n ;The main benefit of the interference source localization is the capability to estimate the position of the interference source, based on the measured angles obtained through the DoA finding techniques defined for jamming and spoofing. In this section, the D&AL approach is considered, but the analysis can be straightforwardly extended to D&CL. The variables defining the model of the problem are:N measurements are available and then two column vectors are added to the defined symbolsN unknown azimuth angles;N estimated azimuth angles.It is assumed that (i) from the same source (aircraft) in different time instants (D&AL), or (ii) from different sources (aircraft) in approximately the same time instant (D&CL).Such measurements are generated either The problem of estimating N angular measurements available, then the above equation becomesn-th column of the Considering The problem has also been solved using Maximum Likelihood Estimation (MLE) assuming that Working with the log-likelihood is more convenient:In order to find the minimum of Equation , where ∇Equation :(10)∇JMLHowever, Equation  does notAs we are looking for the minimum of i and Several numerical solutions have been compared , but the best results are obtained by solving with Levenberg–Marquardt method . This meThe parameter ation in .The techniques for interference detection, localization, and classification can be applied at different stages of the GNSS receiver chain: Front-end  ), pre-conitoring ) or at nnitoring ,26). ThiThe first technique is the so-called AGC detector . This teThe following described techniques are used at the pre-correlation stage too. FPD and TPD  or energy detector) measure the received signal energy over a short period of time; the measured power is then compared with a suitable threshold. The test statistics, in time and frequency domain, are defined in Equations  and 15)15), respThe third detector type is based on the information given by the entropy of the received signal, which is defined as the measure of the average information content per source symbol. The entropy can be calculated asEntropyleThe fourth considered detector is based on the Kurtosis measurement. The Kurtosis measures how much the tails of a distribution differ from the tails of a normal distribution—or, in other words, it identifies whether the tails of a given distribution contain extreme values . Kurtosis is defined asFinally, the last detector described in this paper is based on the Teager–Kaiser (TK) operator . TK measSignal Time-Frequency (TF) Transform: A certain TF transform is applied to the raw signal received by the GNSS antenna. Here, the chosen TF transform is the spectrogram transform, due to its relative low complexity and high accuracy.Image generation: After the TF transform, an image is generated and stored. A library with a huge number of images is created as a training database. The images are labeled and divided as training and testing data sets for further use.Features extraction: Before applying any classification algorithm, an image feature extraction procedure is applied. This is done in order to obtain features from the set of images that can be used to train the algorithm.Algorithm training: The extracted features are used in order to train the classifier. The chosen algorithm was Support Vector Machine (SVM) due to its easy parameter setting and high performance for image classification. The training procedure is called ‘supervised training’, since the images used for training are previously labeled. With this procedure, the algorithm learns which features are related to each interference type.Algorithm evaluation: finally, after the algorithm is trained, it is ready for using testing images (which are not labeled) in order to check the accuracy of the classifier.Besides jamming detection, classification of jamming signals is another important aspect. Not so many efforts have been put in the existing literature so far on the classification compared with detection. In this paper, a solution for jamming classification is also addressed, by using the different features the interference signal introduces in the received GNSS signal. Jamming classification can be split in the following steps :Signal TThe mentioned methods have been applied both in the lab on synthetic signals, as presented in , and on advanced or sophisticated spoofing, is regarded as a high cost, high complexity, and less common type of attack [Many kinds of spoofing attack exist: they differ in the level of complexity/cost of realization at the attacker side, and pose different levels of threat to the target receiver . Amongstf attack . The reaf attack ,31,32: iIndeed, the code and carrier phase pseudoranges produced by the receivers for each Pseudo Random Noise (PRN) code in view , differequations  and 20)i,j is exultipath . In the if more signals share the same geometrical term, they are likely produced by the same source, so they are not genuine (detection);the common DoA of such counterfeit signals can be extracted from the common geometrical term (direction finding).The geometric range difference between the satellite and the antennas gnal DoA:Δdij(m)=gthe carrier phase observables produced at each epoch by three receivers, connected to three antennas properly spaced each other, enter the detection module;the detection algorithm forms the SDs and Double Differences (DDs) for each antenna and signal pair at each measurement epoch; it monitors its detection metric computed from the DD measurements;if a set of signals is declared ‘spoofed’, then the Direction Finding algorithm is activated on the SD code and carrier phase measurements for the current epoch and the DoA is estimated;the SpDDF procedure continues to the next epoch.Taking this into account, it is possible to figure out a procedure which combines the detection of the spoofing and the DoA evaluation (Spoofing Detection and Direction Finding (SpDDF)):The algorithm used for the spoofing detection, named Dispersion of Double Differences ( Squares  improved Squares . A detai Squares . The D3 baseline:∇Δφij(m)=baseline:∇Δφij(m)=quation  consists of a system of Once a subset of shown in  is emploshown in . The forTechnical Institute La Marañosa (ITM) (Spain), a research and development organization belonging to the Spanish Department of Defense and managed by the National Institute of Aerospace Technology (INTA). A car equipped with the technological demonstrator of the concept described so far was driven along two outdoor areas belonging to the ITM Institute. These areas had visibility from the interference source:Location of the interference source (BaseTx). WGS-84 coordinates: Zone Z1. Straight trajectory within 1200 m from the source;Zone Z2. Curve trajectory within 100 m from the source.The methods previously described for jamming and spoofing detection and direction finding have been implemented and validated in laboratory conditions and then in open-field experiments. The open-field experimentation campaign was hosted at the The purpose of the technological demonstrator was to go one step further from the laboratory verification, completing the validation in real conditions  and with real-time hardware acquisition .Interference sources specified for jamming and spoofing attacks were divided in two different types, the main features of which are detailed in The technological demonstrator consists of hardware and software components installed on-board a ground vehicle moving in the areas affected by the interference. The demonstrator has three GNSS active antennas with an L-shaped layout. The preferred configuration for the estimation of DoA in the presence of spoofing are orthogonal baselines whose length is This setup is equal for both jamming and spoofing tests, except for the hardware equipment used for the receiver stage. The receiver stage for spoofing detection is composed of two AsteRx4 GNSS MC/MF modules from Septentrio N.V. (each receiver supports up to two antennas). They share synchronization signals (10 MHz reference clock and Pulse Per Second (PPS) signal) between each other: this configuration is needed to form the synchronized SD and DD carrier-phase and pseudorange measurements.The plan of the test cases execution is summarized in https://zenodo.org/record/3532660#.XekN04jwanY. doi: 10.5281/zenodo.3532660.Combining straight line trajectories with different curve paths and static stages validates all the possible values of the DoA for both spoofing and jamming signals. The raw dataset recorded during trials is available in: The results of processing of the datasets recorded during the live trials in the open field tests are reported in detail in this section. They prove the technological feasibility of a collaborative GNSS interference management concept for a civil aviation scenario, but also highlight the necessity of a deeply integrated design able to cope with the complexity of the system and of the possible scenarios.In the previous step of using the detectors, a threshold Results on jamming detection of on-field data are depicted in For a trade-off refer to ; the worIn order to show a more complete performance analysis of the classification method, Results show that the average accuracy of the detector is more than 98%. Pulsed and Amplitude Modulated (AM) classes are classified with no error after testing the classifier with 2000 images. Narrow Band (NB) class is miss-classified about 1.5% of cases. About 0.5% is classified as interference free, and 1.1% is classified as Chirp jammer . Regarding this, chirp jammer is also miss-classified 0.8% of cases, divided as 0.7% considered as NB and 0.1 as a No jammer scenario. Finally, the Frequency Modulated (FM) testing scenario is miss-classified for about 4% of cases. In addition, 3.8% is miss-classified as an AM jammer , and 0.25% as a Chirp jammer.The quality of the code and carrier phase measurements obtained from the GNSS receivers is critical in the determination of the performance reached by SpDDF. As detailed in ,36, the The layout of the three antennas used for the tests is described in Since the true azimuth of the spoofer location with respect to the vehicle body frame is known in the static tests, a quantitative assessment of the DoA estimation algorithm performance can be performed. Thus, the main performance metrics of the whole SpDDF algorithm are summarized in As a final remark about the static test, it must be noticed that the limited distance of the spoofer with respect to the receiving antennas makes this test configuration very demanding for the PAF algorithm because the DoA of the spoofing signal is not perfectly equal for all the three receiving antennas. Nevertheless, the algorithm was proved to be able to reach an acceptable level of accuracy even in these sub-optimal conditions.According to The This paper presented a new interference management concept able to detect the presence of an intentional interference on the GNSS signals and locate its source. This interference management concept is mainly addressed to aviation applications, where the role of GNSS in the ATC is increasing and the safety risks related to jamming and spoofing attacks raise concerns and special attention. The methods to perform the detection and the localization have been presented for both jamming and spoofing attacks. The analysis of the results obtained during an on-field campaign allowed for assessing the performance of the proposed interference management concept and indicated the future developments that could pave the way for an effective adoption in ATC application.Once the open-field experiments were executed and the results were analyzed and evaluated in accordance with the expected performances, some issues can be highlighted in order to justify the algorithms behavior, mainly: GNSS receiver clock bias estimation for spoofing direction finding and plane wave-front for jamming direction finding. These issues are not related with the algorithm itself, but to the hardware infrastructure used to record the raw input data used by the algorithms and the physical restrictions of the testing area. Taking into account all the limitations identified in the open field experiments and considering some issues that should be investigated still at algorithmic level and laboratory simulations, an evolution of the GNSS interference threats management concept and its benefits should be considered in further investigations.Regarding the direction finding algorithms of jamming and spoofing, some potential benefits could be related to: the computation of the DoA measurement uncertainty, mitigation of error sources during signal acquisition, and reduction of the receiver clock bias. In addition, an optimized choice of the detection threshold and a more robust method to cluster the DD observables should be evaluated in order to improve the spoofing detection performance. Taking into account some of the improvements listed above, the algorithms could be upgraded and validated in new open-field experiments with GNSS interference radiation and longer jamming range to achieve plane wave-front. These experiments would consist of data acquisition and post-processing, as it has been done in the experiments presented in this paper. The definition of a prototype integrating the hardware for jamming and spoofing detection and direction finding, including the software for real-time processing simultaneous for jamming and spoofing, should also be considered for further experimentation campaigns. The improvements identified for the autonomous mode (D&AL) are also valid for the collaborative mode (D&CL). Therefore, it is worth planning future open-field trials to validate the D&CL mode. These trials would be identical to those described in


Diabetic macular edema (DME) and retina vein occlusion (RVO) are macular diseases in which central photoreceptors are affected due to pathological accumulation of fluid. Optical coherence tomography allows to visually assess and evaluate photoreceptor integrity, whose alteration has been observed as an important biomarker of both diseases. However, the manual quantification of this layered structure is challenging, tedious and time-consuming. In this paper we introduce a deep learning approach for automatically segmenting and characterising photoreceptor alteration. The photoreceptor layer is segmented using an ensemble of four different convolutional neural networks. En-face representations of the layer thickness are produced to characterize the photoreceptors. The pixel-wise standard deviation of the score maps produced by the individual models is also taken to indicate areas of photoreceptor abnormality or ambiguous results. Experimental results showed that our ensemble is able to produce results in pair with a human expert, outperforming each of its constitutive models. No statistically significant differences were observed between mean thickness estimates obtained from automated and manually generated annotations. Therefore, our model is able to reliable quantify photoreceptors, which can be used to improve prognosis and managment of macular diseases. In 2017, 425 million individuals were suffering from diabetes worldwide and about 10% of these developed vision-threatening DME1. This number is estimated to increase to 629 million people worldwide by 20453. With RVO, 14–19 million adults are affected globally2. Both conditions are characterised by pathological accumulation of fluid in the sensitive macular layers, severely affecting the central photoreceptors, either causing disruptions in layer continuity6 or abnormal changes in thickness, including irreversible photoreceptor loss7. Functional examinations with microperimetry in other macular diseases have highlighted the topographical association of photoreceptor loss with focal vision impairment8. Particularly when this loss is seen foveally, visual impairment is often irreversible and can significantly affect quality of life8.Diabetic macular edema (DME) and retinal vein occlusion (RVO) are the two most frequent causes of visual impairment in the working age population9 or the amount of intraretinal cystoid and/or subretinal fluid10. In addition, OCT allows to visually assess the photoreceptor layer, which appears as a multi-layered structure with hyperreflective and hyporreflective bands located between the outer limit of the myoid zone  and the inner interface of the retinal pigment epithelium (RPE)  is the state of the art imaging modality for assessing and evaluating retinal changes associated with DME and RVO. OCT scans can be used to quantify relevant biomarkers such as thickness of the central retinal layers15, which have demonstrated to be useful in numerous applications in OCT16. These methods can reduce the burden of the tedious and time consuming manual delineation of the photoreceptors by automatically providing accurate segmentations and allowing to quantify alteration/loss of photoreceptors in large populations as well as individual patients.Such a challenging task can be addressed by the incorporation of computerised models based on artificial intelligence and machine learningSelecting the optimal machine learning model for solving the problem is essential, as certain designs might favour some solutions over others. This behaviour is similar to what we expect from different human observers: depending on their knowledge, expertise and visual skills, they will consider specific characteristics of the structures and decide where the edges of the region of interest are located. A natural way to overcome this issue would be to leverage their diverse opinion and produce a consensus annotation based on a voting strategy by averaging. However, in a clinical setting this is costly—as it requires to work with multiple experts—and extremely time consuming.In this paper we introduce a novel deep learning-based algorithm for automatically segmenting the photoreceptor layers in routine OCT scans. Inspired by the idea of leveraging the opinions of different experts, we propose to train and integrate a series of complementary U-shaped convolutional neural networks architectures. Our ensemble is able to produce not only accurate segmentations, but also consistent photoreceptor thickness maps and disagreement maps that can be used by human experts to identify areas with pathological photoreceptors or errors in the segmentation.40 OCT datasets of 40 patients  were acquired by reading centre-certified masked operators at multiple clinical sites, and raw data was uploaded to the Vienna Reader centre  for subsequent analysis. All scans were captured using a Spectralis OCT device  with the same predefined image protocol, providing volumetric scans composed of 49 B-scans, each containing 512 A-scans, covering an approximate retinal area of 6 × 6 mm. The images were randomly selected before starting the manual annotation process from different clinical studies and from different disease timepoints, to ensure no bias in data selection and enough variability in disease appeareance. Volumes with low quality B-scans were not included as those B-scans were considered non-diagnostic.All patients gave informed consent prior inclusion in the respective multi-centre clinical trials and in- and exclusion criteria were the same for all the patients of one. Both the respective prospective studies as well as this post-hoc analysis adhered to the tenets of the Declaration of Helsinki and the standards of Good Scientific Practice of the Medical University of Vienna. The presented study was approved by the Ethics Committee of the Medical University of Vienna, Vienna, Austria (1246/2016).The automated approach presented in this study is a supervised learning algorithm, which implies that it requires manually annotated images for training. Additionally, ground truth segmentations are needed to compare the outputs of the system with respect to a human expert outcome.17 was applied on each B-scan in our 40 OCT scans database to identify the IS/OS and the OB-OPR interfaces. Subsequently, a group of trained readers manually corrected the delineation curves using an in-house software tool, capturing the area between the top of the IS/OS junction and the outer boundary of the third hyperreflective outer retinal band (OPR/COST/IZ). A senior retina specialist supervised the segmentation process to ensure proper labelling of the region of interest, and corrected the resulting masks when necessary. Whenever this retina expert was unsure about the correct annotation of the region, she consulted up to three additional retinal specialists for independent or group discussion, and a consensus annotation was achieved. In case of low quality A-scans, the experts interpolated their annotation from the neighboring B-scans and A-scans, relying on their knowledge of the particular disease course of the patient.The Iowa Reference Algorithm The 40 OCT datasets were randomly divided into a training, validation and test set, each of them comprising 25, 3 and 12 volumes, respectively. The training set was used for learning all the constitutive models of our ensemble, and the validation set was applied for model selection . The test set was not used until the final evaluation to ensure a proper estimation of the generalisation ability of our approach. In all the cases, a similar proportion of RVO and DME cases was preserved to avoid any disease-based bias.In order to estimate the inter-grader variability, a retina expert (different from the four experts mentioned above) manually annotated a subsample of B-scans randomly selected from the test set. The same protocol used for generating the gold standard labelling was applied, but without any consensus grading and discussion. In particular, three random samples of 2 B-scans were extracted per each OCT volume, each sample located within the regions of an early treatment diabetic retinopathy study (ETDRS) grid at minimum/maximum distance of 0/1, 1/3 and 3/6 mm from the fovea. Such a sampling strategy allows to compare the automated approach and the human observer’s discretion in regions that are affected by the disease at variable levels, without requiring the expert to fully annotate all the 12 volumes.All graders have been trained at one of the largest European retina departments with the largest European reading center for standardized retinal image analysis (Vienna Reading Center). All ophthalmologist from the research group have finalized or will finalize their clinical rotations as retina specialists and have 2–10 years of experience in retinal image analysis besides their regular ophthalmology training.Our segmentation approach is a deep learning method based on an ensemble of U-shaped fully convolutional neural networks (FCNNs). A conceptual representation of these networks is included in the Supplementary Material Fig. 18, the BRU-Net19 and the All-Dropout20 architectures, while the fourth one corresponds to our U2-Net21. The U-Net was selected due to its standard design, whereas the BRU-Net was chosen based on its performance for retinal layer segmentation in pathological OCT scans. On the other hand, the All-Dropout architecture was selected due to its improved generalisation ability, which we hypothesised might help to better deal with healthy areas. These architectures had to be slightly modified to adapt them to this specific segmentation task, always using the validation set to assess the effectiveness of the changes. Our U2-Net did not require any modifications as it is already designed for photoreceptor segmentation in macular diseases21. Detailed descriptions of the architectures, the implementation and the training strategy are provided in Supplementary Material Section   Figure 22 to retrieve a binary segmentation of the photoreceptors.At test time, an unlabelled, full resolution raster scan (512 × 496) pixels was processed by each individual model, recovering four different score maps, one per model, in which a pseudo-probability of being part of the photoreceptors was assigned to each pixel coordinate. The pixel-wise average of these maps is taken in order to obtain a mean score map of the photoreceptors that represents the consensus among the different models. Similarly, the standard deviation is also computed pixel-wise to retrieve a disagreement map across all the automated evaluations. Finally, the average score map is thresholded using the Otsu methodThe binary segmentations and the standard deviation maps are used to estimate en face thickness and model disagreement maps, respectively. The en face thickness maps provide an overall representation of the photoreceptors for the full OCT volume, which can be used to assess the variability in the photoreceptors density or to identify focal disruptions or other pathological changes. Similarly, the en face standard deviation map allows to easily identify areas of disagreement between models, which might be associated either with the presence of morphological pathology or with image artefacts. Moreover, using the en face representation, it is possible to briefly summarise the information of a full volume into a single image.  Figure For the standard deviation, most of the variation is expected to happen on the edges of the layer, which might be interesting for manual correction of the results at a B-scan level. However, we are interested in representing other variations in the pseudo-probabilities provided by each model, such as those occurring inside the photoreceptors area, which might be associated with high uncertainty due to disruptions or concurrent pathology. To this end, the central axis of the photoreceptors is estimated by taking the average position in between the upper and the lower interfaces of the layer. Then, the standard deviation values on these pixel coordinates are mapped to the corresponding position in the en face representation of the standard deviation map.23. Precision (Pr) is defined as the fraction of pixels that were correctly classified as belonging to the area of interest (e.g. the photoreceptors layer) . Recall  accounts for the fraction of identified pixels of interest with respect to the total number of pixels belonging to the region . Pr/Re curves23 are similar to the receiver-operating characteristic (ROC) curves, in which a score map is thresholded at different levels and the resulting sensitivity and specificity values are depicted in a 2D plot. The key difference is that Pr/Re curves plot precision vs. recall values, instead, allowing a better analysis of classification or segmentation results where the class of interest is proportionally imbalanced with respect to all the other elements. In this case, the pixels belonging to the photoreceptors layer represent approximately 2% of the pixels of a B-scan, which results in a suboptimal usage of ROC curves. The area under the precision/recall curve (AUC) quantitatively summarises the performance of the method, with 1 associated to a perfect classification and 0 to a completely inverted result.The proposed approach was quantitatively evaluated in the full OCT volumes and in three different areas of the ETDRS grid: the central subfield (CSF), the 3 central millimetre (CMM) area and the 3–1 CMM ring. The pixel scores provided by the FCNNs were quantitatively evaluated using precision/recall curves24 as twice the product of precision and recall divided by their sum and is an appropriate overall indicator of the quality of the binary results. The mathematical formula including the definition of the Dice coefficient using the relationships between the segmentation and the annotation can be found in the Supplementary Material.The binary segmentations obtained by thresholding the score maps were evaluated in terms of the Dice coefficient, Precision and Recall. Formal definitions of these metrics are provided in Supplementary Material Section n = 12 volumes).The results of the proposed approach were quantitatively compared with the performance of each of its constitutive models using the above-mentioned metrics. Such an assessment allows to study the contribution of model ensembling in improving the results. We studied the statistical significance of the improvements in the segmentation results using one-tail paired Wilcoxon sign-rank tests at a significance level of p < 0.05 . When the assumptions of the t-test were not held , a paired Wilcoxon sign-rank test was used.We also statistically assessed if the differences in the pixel-wise thickness estimation of the photoreceptor layer caused a bias in the prediction of the average thickness. To this end, we used a paired-sample n = 24 for individual samples, n = 72 when using the full sample).The inter-observer variability was assessed using the annotations produced by the second human expert in the sample of B-scans described before. These manual segmentations were compared with respect to the ground truth and evaluated using the Dice coefficient, Precision and Recall. By contrasting these results with those obtained by the proposed ensemble, we can study if the performance of the algorithm is in line with the one of a human expert, as evaluated using the same ground truth annotations and the same metrics. The statistical significance of the differences in performance was assessed using one-tail paired Wilcoxon sign-rank tests at a confidence level of 0.05 , which indicates that the errors of the automated approach did not affect the average thickness estimation.The average thickness of the photoreceptors was computed for each area of the ETDRS grid based on the thickness maps calculated from the ground truth annotations and from the segmentations provided by our ensemble. Independently of the area of analysis, we observed that the differences in the average thickness values were not statistically significant  and manually engineered features. Alternatively, we have presented a deep learning approach that is able to properly identify the area of interest not only in healthy regions, but also in areas with variable alterations due to the underlying disease such as RVO or DME. The only available deep learning method which is successful in pathological OCT scans was our own U2-Net model, recently published21.In this study, we have introduced an automated deep learning method for segmenting and quantifying the photoreceptors in OCT scans of patients with DME and RVO, diseases with significant disturbance of intraretinal morphology. Only an automated annotation of photoreceptor integrity and its thickness in particular, can reliably identify the level of photoreceptor alteration in its current status and during follow-up. Few attempts were made in the past, yet all of them differ from ours in several key aspects. Chan 32. This is of major importance when clinical decisions have to be made during early diagnosis and long-term follow-up and an individual prognosis of disease severity and/or therapeutic outcomes is needed.Our automated approach is built on the concept of multi-grader averaging. To identify the photoreceptor layer manually, certified experts annotate the regions of interest. Then, the average of these annotations can be taken as a more realistic representation of the photoreceptor layer condition. Typical OCT artefacts such as noise, saccadic motions and vessel shadows might be misinterpreted by human readers as layer disruptions, while certain bands such as the interdigitation zone (IZ) might not be easily separated from the RPE, leading to inter-expert variability. Furthermore, as previously mentioned, abnormal cases with multiple concomitant lesions such as intra- and subretinal fluid are much more difficult to interpret and annotate than healthy scans. The proposed algorithm is able to alleviate this issue. Our human-inspired hypothesis ensembling the outputs of different fully convolutional, U-shaped neural network architectures allows to achieve consistently better results than using each model individually and has been confirmed during extensive evaluation on 12 comprehensive OCT volumes including a total of 588 B-scans. Our approach provides not only an accurate morphological segmentation of the photoreceptor band, but also gives qualitative feedback about the disagreement between its constitutive models, allowing experts to realistically identify potential areas of errors. Furthermore, fully-automated approaches usually have a much higher intra-grader reproducibility, whereas the manual task of segmenting retinal structures does not offer high reproducibility valuesThe data applied both for training and evaluating the proposed approach correspond to OCT volumes collected from several clinical studies, meaning that their quality is overall in line with the one expected from clinical routine examinations. nevertheless, anatomical and disease related artifacts were rather common in this highly pathological data. In particular, Fig. A substantial inter-observer variability was observed in manual photoreceptor annotation, particularly in scans with extensive macular edema like the one shown in Fig. The interpretation of our method is also much better explainable to clinicians than a stand-alone black-box segmentation output of a single model. We have proposed to automatically compute model disagreement maps (standard deviation between the score maps provided by each constitutive model of the ensemble) that can be used to carefully analyse specific B-scans. Highlighting areas of disagreement between the automated observers in an en-face fashion indicates potential errors in their individual segmentations. We also observed that most of the variabilities are explained by the true presence of pathologies (e.g. disruptions or thickening of the photoreceptors). Therefore, it is more likely that the disagreement maps highlight areas where disease activity or photoreceptor loss has occurred and is responsible for vision loss. Regions where all segmentations are the same and non-questionable correspond more reliably to healthy areas of the photoreceptor layer, as we observed in our experiments. In few cases, the disagreement occurred due to image artefacts such as vessel shadowing, which is easily recognised by the clinician when the maps are presented together with an en-face retinal image containing the vessel structures (e.g. the fundus image taken together with each OCT volume).33 or genetic dystrophies such as retinitis pigmentosa34, or for estimating other morphological properties such as the overall length of photoreceptors associated with several diseases, including DME29 and RVO28 or even early age-related macular degeneration. Our automated photoreceptor segmentation could be prospectively applied in clinical studies and allows for the first time to segment the photoreceptors on each individual voxel of the raster scan instead of using random point thickness measurement obtained manually. The method can also allow the interpretation of retrospective studies in which pathognomonic changes in photoreceptor morphology are assessed during pharmacological treatments12.We have also presented a strategy for automatically computing en-face thickness maps of the photoreceptors. This approach is also applicable to study the effect of pathological changes in photoreceptor thickness due to other diseases such as glaucomaet al. used an ensemble of neural networks for brain tumour segmentation that reached the first place in the MICCAI-BRATS contest35. Similar ideas were recently explored in other OCT imaging-based tasks such as identifying geographic atrophy lesions36 or segmenting artefacts37.Leveraging the outputs of different models reproduces the idea of conclusively integrating the judgement of multiple experts, although it is more efficient and less expensive. Despite the fact that this principle is not novel scientifically, not many studies have taken advantage of this in the past, and none of them for photoreceptor segmentation. In particular, Kamnitsas The proposed approach is sufficiently general to incorporate any deep learning model for segmentation. The decision for the proposed networks was based on its popularity at the time of use and the judgement of the authors to obtain useful results for this segmentation task. The selected networks seem to be appropriate, according to the results. The All-Dropout network achieved high precision but low recall, meaning that it is unable to retrieve some pathological photoreceptors. The opposite behaviour was observed in the BRU-Net, which produced segmentations with a larger number of false positives. The classical U-Net performed similarly in the different evaluation areas, although with higher precision. Compared to all others, the U2-Net achieved better results, which is consistent with the fact that it was especially designed for photoreceptor segmentation in RVO, DME and AMD. In this paper we build on top of our previous contribution (U2-Net) by integrating the outcomes of other artificial intelligence approaches that were specifically adapted to the challenging task of photoreceptor segmentation. By means of this approach, we were able to achieve better results and exploit the individual advantages of each network, taking automated feature segmentation at the level of the retina a large step further.In conclusion, we have introduced a deep learning-based convolutional neural network ensemble for automatically segmenting and quantifying the photoreceptors in routine OCT images of patients with DME and RVO. This achievement is most relevant in research and clinical routine as it offers a reliable interpretation of the photoreceptor condition for each individual and at each time. Identification of photoreceptor alteration may strongly improve our insight into the pathophysiology of retinal/macular disease and allow a quantification of functional loss based on true morphology.Supplementary information