Regulatory Circuits project is one of the largest efforts for integrating human cell genomics data to predict tissue-specific transcription factor-genes interaction networks. In spite of its success, it exhibits the usual shortcomings limiting its update, its reuse , and its extension with new data samples. To address these limitations, the resource has previously been integrated in an RDF triplestore so that TF-gene interaction networks could be generated with two SPARQL queries. However, this triplestore did not store the computed networks and did not integrate metadata about tissues and samples, therefore limiting the reuse of this dataset. In particular, it does not enable to reuse only a portion of Regulatory Circuits if a study focuses on a subset of the tissues, nor to combine the samples described in the datasets with samples from other studies. Overall, these limitations advocate for the design of a complete, flexible and reusable representation of the Regulatory Circuits dataset based on Semantic Web technologies.In life sciences, there has been a long-standing effort of standardization and integration of reference datasets and databases. Despite these efforts, many studies data are provided using specific and non-standard formats. This hampers the capacity to reuse the studies data in other pipelines, the capacity to reuse the pipelines results in other studies, and the capacity to enrich the data with additional information. The Linked Extended Regulatory Circuits (LERC). It consists in (i) descriptions of biological and experimental context mapped to the references databases, (ii) annotations about TF-gene interactions at the sample level for 808 samples, (iii) annotations about TF-gene interactions at the tissue level for 394 tissues, (iv) metadata connecting the knowledge graphs cited above. LERC is based on a modular organisation into 1,205 RDF named graphs for representing the biological data, the sample-specific and the tissue-specific networks, and the corresponding metadata. In total it contains 3,910,794,050 triples and is available as a SPARQL endpoint.We provide a modular RDF representation of the Regulatory Circuits, called Regulatory Circuits datasets and facilitates its reuse in other studies.The flexible and modular architecture of LERC supports biologically-relevant SPARQL queries. It allows an easy and fast querying of the resources related to the initial https://regulatorycircuits-lod.genouest.org Limits to data reusability in life sciences In life science, there has been a long-standing effort to standardize and integrate reference datasets and databases [atabases , 2. Despatabases . This haatabases .The Regulatory Circuits project The Regulatory Circuits project [ project , 6 is on project  and ENCO project , and areRegulatory Circuits original resource is the lack of intermediary results: only the input files (some with pre-processing) and the resulting tissue networks are accessible. The resource does not describe the specific contributions from the different samples nor the specific contributions from the enhancers and the promoters in the tissue networks. Another drawback is the lack of reproducibility of the pipeline that leads to the impossibility to compute those intermediary networks. As the regulation scores in each tissue-specific network depend on the weights of the enhancers and the promoters in all the samples, these two drawbacks make it impossible (i) to update Regulatory Circuits along the successive releases of the resources it is based on, (ii) to reuse only a portion of Regulatory Circuits if a study focuses on a subset of the tissues, and (iii) to combine the Regulatory Circuits samples with samples from other studies.The main drawback with the Semantic Web technologies Semantic Web technologies have long been perceived as a relevant framework for supporting data integration and reuse [nd reuse , and havnd reuse , 11, butnd reuse . This isnd reuse .RDF data structure for the Regulatory Circuits project. In a previous study [Regulatory Circuits resource, consisting of 3,226,341 entities and 335,429,988 relations between them. As an application case-study, we demonstrated that TF-gene interaction networks through promoter and enhancer for each cell sample could be generated on-the-fly with two SPARQL queries. Those two queries are time efficient to retrieve the TF-genes relations of enhancer sample-specific networks and promoter sample-specific. However, those queries do not combine the scores from enhancer and from promoter nor combine the samples constituting a tissue to obtain its regulatory network. The solution proposed does not store the computed networks and requires to be computed each time a user may need them. Another limitation in the use of this resource is the lack of metadata. These are necessary to identify the subset of the samples that meet a user’s specific criteria when reusing Regulatory Circuits. Overall, these limitations require a more complete, more flexible and reusable representation of the Regulatory Circuits dataset.us study , we provExpected benefits of RDF technologies In this article, we elaborate upon the strategy of [Regulatory Circuits source biological data , linkedApproach First, we designed the structure of the sample-specific graphs as well as the SPARQL queries for computing the weights associated to TF-genes relations. Second, we designed the structure of the tissue-specific graphs as well as the SPARQL queries for computing the weights and scores associated to TF-genes relations based on the values computed for the samples. Third, we described the biological data associated to the samples and the tissues, we complemented them with mappings to external public databases, and used them to enrich the original dataset with an experimental context graph. Finally, we proposed a modular organization of the aggregated datasets into RDF named graphs linked by an additional metadata graph, which allows to identify the relevant portions of the dataset in order to maintain query performances.Linked Extended Regulatory Circuits (LERC), a flexible resource to query tissue- and sample-specific TF-genes interactions. We provide an RDF representation of the Regulatory Circuits, called Linked Extended Regulatory Circuits (LERC). It consists in (i) descriptions of biological and experimental context, linked to the references databases, (ii) annotations about TF-gene interactions at the sample level for 808 samples, (iii) annotations about TF-gene interactions at the tissue level for 394 tissues, (iv) metadata connecting the three above listed named graphs. Overall, the resource contains 3,910,794,050 triples and is available as a Virtuoso endpoint at https://regulatorycircuits-lod.genouest.org/ [Regulatory Circuits’s final results in RDF.est.org/ . We showLERC is applicable to any similar dataset produced in other projects.This integration scheme used to construct named graphs was used to link all the RDF datasets together. This modular design allows (1) to assign metadata describing each group, (2) to improve SPARQL queries performances by only considering some relevant portions of a dataset, (3) to extend the dataset by adding new groups such as new samples data and (4) to reuse some portions of the dataset in other studies. Note that this addresses the limits to data reusability identified at the beginning of the All the results presented in the paper were obtained by relying on Semantic Web technologies. Our strategy was to create an RDF graph for each different dataset handled in the regulatory circuits project . Then, the capability of the RDF language to identify groups of related triples as Regulatory Circuits website and supplementary data give access to unstructured, disconnected and diversely formatted tabulated files related either to input biological data  or computation intermediate results (59 files). The main output of the in silico analyses resulting from the Regulatory Circuits project consists in maps  describing interactions between TFs and their target genes in each of the 394 studied tissues.The Each network is described by an oriented graph in which TFs are connected to genes. The nodes are annotated with biological information (gene IDs for both TFs and target genes). The edges are annotated with a unique score aggregating two different weights representing the respective contributions of the enhancer and promoter regions to the predicted strength of the TF-gene regulation. These respective contributions as well as the formula used to compute the final score are neither described nor available. These 394 tissue-specific TF-gene interaction networks are provided as tabulated files, and the pipeline to produce them is neither usable nor reproducible.Regulatory Circuits networks together with their attributes , and describes the relations between these entities. As detailed in [The biological data graph contains the minimal set of biological entities required to build the ailed in  and depiGene), transcripts  and transcription factors (691 instances of the class TF) are constructed based on the names provided by the Regulatory Circuits datasets . These identifiers are linked to identifiers from external databases such as UniProtKB [Regulatory Circuits datasets.The identifiers of genes  and Enhancer .There are two classes of regulatory regions: confidence of transcription factor binding site in the region , two between regulatory regions and transcripts weighed by the distance and the Weight_Distance between those entities , and a last one between transcripts and genes . Each instance of classes Promoter or Enhancer is associated with two sets of 808 float values, one corresponding to its expression value in each sample, and the other corresponding to its normalized relative rank in each sample compared to the 807 others. Similarly, each instance of the Transcript class is associated with 808 float values, describing its normalized relative rank in each sample compared to the others. This rank information is directly provided by Regulatory Circuits. Contrary to the promoters and enhancers, no measured expression value is provided for transcripts. For the LERC resource, the ranks were computed according to the methodology described in [libId).The dataset comprises five types of reified relations: two between TFs and regulatory regions weighed by the Figure confidence value, and the rank of the regulatory region is described by a value associated with the sample. The promoter weight is defined by Each TF-gene interaction is characterized by a promoter weight and by an enhancer weight. As shown in Fig. weightP is given in Fig. SAMPLE must be replaced by the identifier of an actual sample. A similar query for computing weightE is available on the GitHub repository of the project  and integrated the final 808 sample-specific networks in our resource triplestore, by using an INSERT operation as shown in Fig. Max score combining the two previous one, and (iv) a RC score extracted from the Regulatory circuits output data files. We designed a SPARQL query to compute tissue-specific promotor/enhancer weights and MAX scores and re-inject them into the tissue-specific RDF graphs  a promoter weight (max of the promoter weights among the samples composing the tissue), (ii) an enhancer weight (max of the enhancer weights among the samples composing the tissue), (iii) a phs Fig. . It compexperimental context graph describes the experimental information about the 808 samples , about the 394 tissues (linked to the samples they are composed of) as well as the mappings to reference databases. Note that the experimental data (expressions and ranks) belong to the biological data graph. All the information contained in the experimental context graph are extracted from the nmeth_.3799-S2.xlsx file present in the Regulatory Circuits supplementary data, and formatted to respect the identifiers of the generated samples-specific or tissue-specific RDF graphs.The When applicable, we also include links to other knowledge bases from the Linked Open Data such as gene identifiers from Ensembl, protein identifiers from UniProtKB, cell types and anatomical structures from the Uberon and the Foundational Model of Anatomy ontologies.metadata graph contains all the information about the other graphs including their VoID descriptions, as well as the associations of the samples and tissues from the experimental context graph with their respective graph containing their specific regulatory network.The This explicit representation of the metadata about the samples and the tissues can be queried by the users for identifying the subset of samples or tissues they are interested in. The modular approach described next allows the user to retrieve the corresponding portions of the dataset.Regulatory Circuits that makes it possible to identify the subset of the samples and tissues that meets the user’s requirements, to retrieve the corresponding networks and to combine them with additional data. To do so, we first created an RDF named graph for general biological data such as the binding and neighborhood relations between TFs, regulatory regions and genes. Second, we created a distinct RDF named graph for each sample- and tissue-specific network  resource are available on the website of the project: https://regulatorycircuits-lod.genouest.org. The RDF version of the dataset is under the Attribution 4.0 International (CC BY 4.0) license. The SPARQL queries used to generate the sample and tissue-specific TF-gene graphs are available on GitHub https://github.com/mlouarn/RCsparql/. The generated turtle files are available at https://zenodo.org/record/4889146.All data related to Regulatory Circuits published methodology, the TF-gene interactions are mediated by the ability of the TF to bind into regulatory regions of the chromatin (enhancers or promoters), the distance of this region to the gene (enhancer being farther and promoter being adjacent to the genes), the region accessibility and the gene expression. Each TF-gene interaction is therefore characterized by a promoter weight and by an enhancer weight  and (ii) an enhancer weight (class WeightE). They are obtained as the maxima of the corresponding weights of the same relation in the RDF graphs specific for all the samples constituting the tissue.As described in the Regulatory Circuits project, each tissue is associated with 1 to 33 samples. It is therefore relevant to build TF-gene regulatory networks at the tissue scale by aggregating the information provided at the sample scale. A tissue-specific graph is the result of a SPARQL query with a UNION pattern on the source biological data (to identify TF and genes entities) and all the sample-specific networks (among the 808 described in the previous paragraph) associated with the considered tissue. As in sample-specific networks, TF-gene relations are first characterised by (i) a promoter weight  may need to be combined in a single score. In this direction, the Regulatory Circuits project provided a global score for each TF-gene interaction in a tissue, which was integrated in our resource as the RC score. Unfortunately, no information is available on the formula used to compute this score. In order to gain in flexibility, we introduced the possibility to enrich the resource with additional aggregating scores: for instance, we computed the maximum of the two weights weightP and weightE, integrated in the resource as the Max Score  and we have no explanation for the TF-genes interactions that have a positive score according to Regulatory Circuits, even though the corresponding scores in all the associated samples are zero.From the RDF graph, we considered that a TF-gene interaction was present in the tissue if the score aggregating promoter and enhancer weights was strictly positive. According to the scores extracted from All the TF-genes interactions, with their weights and scores, computed or from the original networks are stored in their respective tissue-specific named graphs and are available for querying following the SPARQL patterns available on GitHub.metadata graph, which contains all the information about the other graphs including their VoID descriptions and characteristics about the graphs , as well as the associations of the samples and tissues with their respective graph. The RDF model is supported by the regulatorycircuits.owl ontology provided on the GitHub repository , 1 experimental context graph (in purple), 808 sample-specific graphs (in green), 394 tissue-specific graphs (in orange) and 1 metadata graph (in grey). Their main characteristics are as follows: biological data graph representing the biological data of the Regulatory Circuits and FANTOM5 projects was already published in [The source ished in   and its associated scores or weights, using its associated RDF graph.Using two different sample-specific graphs (or tissue), the query compute_network_2samples.rq enables to extract the union of the scored relations TF-genes for two samples (or tissues).On a sample or tissue named graph, the query find_targets.rq enables to find the targets of a specific TF in this sample (or tissue).Using two sample or tissue-specific RDF graphs, the query compare_targets.rq compares the targeted genes of a specific TF across two different tissues. It gives the union of the TF-genes relations and allows to compare their scores in the two tissues.The query compare_regulator.rq finds the different weights of the common regulators of a set of genes in two given tissues using two named graphs (tissue or sample specific).The query get_tissues_info.rq retrieves the samples composing a tissue, the biological tissue, age of the patients and other experimental information, using the experimental context graph.The query Limit_exp_conditions.rq limits the experimental conditions by retrieving the names of the samples composing a tissue that follow some restrictions: for example selecting only the samples from patients who are over 55 years old.The query Limit_to_healthCondition.rq lists all the samples composing one biological tissue with a disease or an health condition (non specific) associated and the name of the condition.The query Limit_to_healthy_samples.rq lists all the samples composing one tissue that are not linked to any disease.The query Samples_specific_disease.rq lists all the samples that are linked to a specific disease or diseases containing one word.The queries query-samplesInAnatomicalLocation.rq, query-samplesInCellType.rq and query-samplesInDisease.rq are federated queries that leverage the mappings to Uberon, the Cell Ontology and to the Human Disease ontology to retrieve the samples that match criteria requiring some ontology-based reasoning.Exploring LERC in terms of flexibility to tailor analyses according to the needs of the user. For example, let us consider the tissue-specific TF-gene interaction graph “CD14+ Monocytes” which contains 2,402,498 scored TF-genes relations in LERC. However, we notice that the tissue is composed of 33 samples. Among these samples, only 3 were measured in blood , in men aged 47, 57 and 53 (result of the query get_tissues_info.rq). If the user is specifically interested in CD14+ Monocytes from blood for patient over 55, the experimental context graph can be queried with the query Limit_exp_conditions.rq. The query returns a unique sample respecting the conditions. The user can then use the query extract_network.rq with this sample. The resulting TF-gene interaction graph contains 846,930 interactions, pointing out the putative specific effect of genes on this tissue and patient category. We notice that such a tailored result is obtained with a very short SPARQL query  may need to be combined in a single score.For the sake of further studies and to follow the original Regulatory Circuits project [In this article, we address the issue of making the biological datasets from the  project  reusableLERC resource to represent and extend the Regulatory Circuits project as an RDF dataset follows the best practices, using reification for weighted relations, and using named graphs to link the RDF graphs created for each different dataset handled in the regulatory circuits project. It also provides mappings to reference databases such as UniProtKB, Ensembl, Uberon, the Foundational Model of Anatomy ontology (FMA) and the Human Disease Ontology and follows the faldo chromosomal localization format. Federated SPARQL queries can then be used to combine information for Regulatory Circuits with information from these resources . The LERC resource is available on a persistent domain and all queries are publicly available on GitHub.According to the Findability, Accessibility, Interoperability, and Reuse (FAIR) guidelines , the conRegulatory Circuits dataset into RDF with our modular principles, and by allowing flexible SPARQL queries on the LERC resource, our contribution to reusability is threefold. First, it facilitates the reuse of Regulatory Circuits results in other studies by providing access to the tissue-specific regulatory networks and the associated information. Second, it facilitates the reuse of the studies’ data in other pipelines by providing access to the samples’ experimental context and to the intermediary results such as the sample-specific regulatory networks, which can be reused to compute other indicators than Regulatory Circuits published weights and scores. Third, it provides the capacity to enrich the Regulatory Circuits dataset with additional information as the data model and Semantic Web technologies support adding new samples or defining new tissues, and the SPARQL queries we provide can generate the corresponding weights and scores. Overall, the Regulatory Circuits case study confirms that Semantic Web technologies are a relevant solution for reusing knowledge bases [By both converting and enriching the ge bases , 19, andge bases .Improving the exploration ofRegulatory Circuits’s biological data and networks In a previous work [Regulatory Circuits workflow can be described using Semantic Web technologies thus increasing its reproducibility. This new implementation, including not only the input data but also the TF-gene interaction networks resulting from the in-silico integration of source biological data, allows a more flexible (re)use of Regulatory Circuits. The implementation we propose allows a fine-grained exploration: the user can select portions of the network, for example excluding regions at a lower distance than the Regulatory Circuits threshold, or excluding one type of region (e.g. for taking into account that promoters relations are more reliable).ous work  we showeImproving the reusability and the enrichment of the source biological data with SPARQL queries The networks available in the Regulatory Circuits website are static and cannot be updated when the biological datasets it was based upon evolve. A major advantage of our approach is that TF-gene interaction networks for samples and tissues are generated with SPARQL queries from the source biological data before being inserted in the resource. In addition, this implementation also allows the user to easily change some parts of the pipeline that generates the network, such as new calculation of the ranks, to add new genes or transcription factors in the networks, or to remove some of them. All these changes can easily be implemented by adapting the few available queries used to generate the network but necessitate to re-compute the new networks.Regulatory Circuits project. Regulatory Circuits presents 394 tissue-specific networks, but looking into the detail of the samples and tissues revealed that some tissues could be separated into smaller sets. For example, the “CD14+ Monocytes” network given in Regulatory Circuits is based on 33 CD14+ monocytes cell samples, which have different characteristics . The modular structure of our resource allows for the computation of new TF-gene interaction network using these characteristics to better discriminate the samples.Our resource and the approach we use to populate it also facilitates the generation of new TF-gene interaction networks for new tissues, through the aggregation of samples with characteristics different from those chosen in the LERC introduced in this article and the strategy used to build it allows the user to add new tissues or TFs if they have similar input data. This would require to pre-compute rankings for transcripts and regulatory regions which are at the moment provided by the Regulatory Circuits resource and cannot be recomputed. Similarly, introducing a new TFs would require to introduce new confidence values for their binding to regulatory regions.Finally, using the Improving interoperability Among the 217 articles citing Regulatory Circuits, at least 42 either use directly the resulting networks for biological data explanation or use them as comparison for regulatory network inference. In 10 of these, Regulatory Circuits was used in combination with one or several other databases. Other resources on TF-genes relations exist [Regulatory Circuits, the latter being the only one categorizing tissue-specific networks. By representing Regulatory Circuits as an RDF graph we therefore improve its interoperability with resources of similar scope already based on Semantic Web technologies, which facilitates its reuse in combination with other already existing RDF resources. In particular, it significantly extends the portion of FANTOM5 data available as RDF [ns exist  but are e as RDF , 24.We present a generic approach for the enrichment of source biological data with the result of data analyses. Our results show that a Semantic Web approach scales not only for the integration of large-scale biological data but also for the iterative enrichment of such a resource with the results of in-silico analyses modeled with SPARQL queries. This is possible by using a modular structure based on RDF named graphs. This strategy could be easily transposed to other large scale life science studies which analysis pipeline describes relations with simple arithmetic functions.


In this study, three cases involving patients who required multidisciplinary treatment for the aesthetic and functional rehabilitation of the maxillary or mandibular arch are described. In particular, an indication for preprosthetic orthodontic treatment, such as orthodontic extrusion, tooth realignment, correction of malocclusion, and diastemata closure, was present in all cases. Preprosthetic orthodontic treatment to resolve these issues before the restorative procedures was proposed; however, all patients refused preprosthetic orthodontic treatment. Thus, to restore aesthetics and function, solely a feather-edge prosthetic protocol has been used. The biologically oriented preparation technique was used to prepare the teeth that were moved in the established direction by preparing the abutment more on one side than the opposite. This so called “prosthetic orthodontic approach” allowed resolving clinical issues that would typically require preprosthetic orthodontic treatment, such as complete clinical crown loss, occlusal vertical dimension loss, tooth misalignment, malocclusion, tooth agenesis, and severe multiple diastemata. The degree of reciprocal movement of the prepared teeth achievable through this approach was minor and not comparable to a traditional wide-range orthodontic movement. Besides, the technique resulted in a modification of the gingival tissues and improvement of their thickness although it is unclear what effect this technique has on the gingival biotype. None of the patients had prosthetic or periodontal complications for at least 12 months following the procedure. Gingival health was excellent, and the prosthetic procedure did not affect the pulp survival of the vital teeth. The biologically oriented preparation technique used with a prosthetic orthodontic approach can effectively manage complicated cases without the need for preprosthetic orthodontics. The biologically oriented preparation technique (BOPT) is a prosthetic protocol which consists of a feather-edge subgingival preparation whereby the gingival profile adapts itself to the new prosthetic coronal emergence profile . This isThe BOPT protocol has been shown to increase gingival thickness and achieve stable and healthy soft and hard tissues in natural teeth , 4. The An example of soft tissue adaptation to the new prosthetic emergence profile after the BOPT protocol is shown in Figures The prosthetic procedure, laboratory stages, and used materials were the same for all three clinical cases described below.The preparation technique we used in all cases followed the BOPT protocol described by Loi in 2013. This technique involves the use of a diamond flame-shaped bur  for the abutment tooth preparation mounted in a high-speed handpiece. Before starting the dental preparation, the gingival sulcus was sounded with a calibrated probe  to establish the preparation depth. The tooth was prepared until the bottom of the probed sulcus. A vertical area without a finish line was created, and the interim restoration was relined using a self-curing methacrylate resin . The new emergence profile was positioned in the gingival sulcus without violating the biologic width. The movement of the crown axis of the abutment teeth was achieved by means of the BOPT with the POA in all cases where tooth movement was required. The tooth movement was obtained by preparing the abutment teeth with the feather-edge preparation (BOPT) more on the one side than the opposite. For example, if mesial movement of the abutment tooth was required, the tooth was prepared more on the distal side so that the abutment tooth could be moved mesially . In thisA 71-year-old man, American Society of Anesthesiologists (ASA) type I, presented to a hospital dental department hoping to correct aesthetic and functional problems of the maxillary arch  where thA 70-year-old healthy woman presented at a private practice dental office, unsatisfied with her smile; this was due to excessive maxillary gingival display (a gummy smile), concomitant with short clinical crowns . IntraorDuring the next appointment, the anterior maxillary teeth were prepared with the BOPT, and the interim restoration was cemented. Next, two 3.8 mm diameter implants  were placed in the area of the mandibular first molar and left submerged during the osseointegration period. At the end of the session, two alginate impressions and a new centric relation record were obtained and mounted in an articulator to fabricate a mandibular full-arch restoration corresponding to the increased OVD.The mandibular full-arch restoration was fabricated using a 3 mm OVD increment, and the curve of Spee was harmonized with the opposing arch. The mandibular teeth were prepared with the BOPT, and the complete restoration was placed in the mandibular arch from the right to the left second molar. The mandibular left second molar was mesially inclined, so it was prepared more on the mesial side than on the distal one using the BOPT with the POA to move the dental axis distally, thereby creating an adequate space for the first molar parallel to the path of prosthesis insertion . The POAA 63-year-old man, ASA type I, presented at a hospital dental department. The subject wanted to improve their dental aesthetics and function. Intraoral examination revealed a 6 mm maxillary midline diastema and multiple diastemata, rampant caries, radicular fractures, and periodontal involvement of the first molars, which were immediately extracted and replaced with 3.8 diameter implants  that were loaded 3 months after placement .The correct tooth proportions were achieved by preparing the distal side more than the mesial side with the BOPT and POA. Thus, during abutment preparation, the vertical area was moved toward the centre of the tooth, and the mesialisation of the teeth concurrently facilitated diastema closure and maintained the correct tooth proportions . As prevEight weeks after soft tissue maturation, the gingival tissues appeared healthy , and theThe BOPT provides a feather-edge subgingival preparation on natural teeth, for which good clinical periodontal outcomes and thickening of keratinized tissues are reported –6. FeathThis abutment tooth preparation consists of a vertical area in which the crown margin is positioned. This vertical area can be moved in a given direction by preparing the abutment tooth more on the one side than the other, and the crown restoration is moved together, allowing the clinician to move the tooth axis in a mesio-distal or buccal-lingual direction . These cThis prosthetic procedure allowed for the transformation of the tooth shape and position, as is presented in the first case report in which the maxillary canines were transformed into the missing lateral incisors and the first maxillary premolars were moved mesially and transformed into canines. In the other cases, the BOPT with a POA was able to facilitate significant improvements in function and aesthetics in patients with a collapsed OVD with a limited prosthetic space, tooth misalignment, gingival recession, a gummy smile, and multiple and severe diastemata. This procedure was adopted in these cases because tooth wear and aesthetic problems were also present. The vertical area without a finish line, in contrast to the horizontal finish line, allowed clinicians to move the interim restoration in an apical or coronal direction, which greatly facilitates tissue adaptation to the emergence profile of the crown restoration –6. In thOrthodontic treatment with a multidisciplinary approach would have been recommended as the primary treatment option for all of the clinical situations described in this study; however, in cases in which a patient is unable or unwilling to undergo preprosthetic orthodontic treatment, it is possible to instead use the BOPT with a POA to simulate multidisciplinary treatment –24. ThisIn the present case report, the BOPT used with a POA effectively managed complicated cases without the need for preprosthetic orthodontic treatment. The absence of a finish line in natural abutment teeth offers many possibilities to clinicians for solving situations in which a multidisciplinary approach is usually required. After at least one year, none of the patients experienced prosthetic or periodontal complications; the gingival tissues were healthy, and the odontotomies did not affect the pulp survival of the vital teeth. Nevertheless, long-term clinical studies are needed to validate the results of these cases.


Basal septal hypertrophy (BSH) is an asymmetric, localized thickening of the upper interventricular septum and constitutes a marker of an early remodelling in patients with hypertension. This morphological trait has been extensively researched because of its prevalence in hypertension, yet its clinical and prognostic value for individual patients remains undetermined. One of the reasons is the lack of a reliable and reproducible metric to quantify the presence and the extent of BSH. This article proposes the use of the curvature of the left ventricular endocardium as a robust feature for BSH characterization, and as an objective criterion to quantify current subjective ‘visual assessment’ of the presence of sigmoidal septum. The proposed marker, called average septal curvature, is defined as the inverse of the radius adjacent to each point of the endocardial contour along the basal and mid inferoseptal segments of the left ventricle.Robustness and reproducibility were assessed on a cohort of 220 patients, including 161 hypertensive patients (32 with BSH) and 59 healthy controls.The results show that compared with the conventionally used wall thickness metrics, the new marker is more reproducible  and better correlates to the functional parameters related to BSH, with main difference (absolute rank correlation 0.417 vs. 0.341) in local deformation changes assessed by longitudinal strain.Average septal curvature is a more precisely defined and reproducible metric than thickness ratios, it can be fully automated, and better infers the functional remodelling related to hypertension. The increase in cardiac wall thickness in hypertensive heart disease is gradual and not uniform . In earlThe link between this localized thickening and elevated blood pressure was demonstrated when volunteers with basal septal hypertrophy (BSH) and no known history of hypertension were diagnosed with masked hypertension using 24-h blood pressure monitoring . In the This morphological trait is thus an early sign of structural and functional remodelling , and hasApart from the quantitative wall thickness measurements, an equally important diagnostic criterion is the visual assessment of the septal geometry as the shape of the BSH is also distinctive. This diagnostic marker was termed a sigmoidal septum or an upper septal ’knuckle’ ,8,15,16.The problem addressed in this article is this lack of a clear criterion and reproducibility in the assessment of BSH. Together with the variability in the characteristics of the examined cohorts, this problem causes current difficulties in determining the prevalence and the associated risks of BSH. The solution proposed to distinguish between patients with and without BSH in a robust and reproducible manner is a combination of a semiautomatic segmentation of the left ventricle (LV) and the computation of curvature along the LV endocardium of the septal wall. The rationale is that curvature is a metric that objectively quantifies the subjective visual assessment of the presence of the sigmoidal septum, and thus removes human variability. The motivation is also based on the hypothesis that the curvature of the endocardial contour is more reproducible than metrics relying on highly variable thickness measurements.A retrospective study of hypertensive patients is presented here in order to evaluate the performance of the septal curvature compared with the traditional methods. We analyse the intra-observer and inter-observer variability of the metrics and validate the hypothesis of the improved reproducibility of curvature. We then investigate the relationship of the anatomical BSH metrics with early markers of functional remodelling associated with hypertension, finding that curvature is a better descriptor of the deformation changes. We finally explore the relationship between thickness and curvature, both by their correlation and by the diagnostic agreement based on defined thresholds, and discuss the clinical relevance of our findings.N = 161) with well-controlled blood pressure, treated with antihypertensive drugs for a minimum of 3 years, were included. Patients were recruited from the outpatient clinic and general practitioner referrals. Exclusion criteria included history of heart failure or previously known target organ damage. Volunteers from the local community (N = 59) who were presumed healthy, without prior history of hypertension, diabetes or other significant cardiac or noncardiac diseases represented healthy controls. Full description of the cohorts is included in the supplementary material.Hypertensive patients  or parasternal long-axis (PLAX) view . In ordeCurvature is the amount by which a line deviates from being straight; therefore, it is expected to be highest around the apex, and low at the mid and basal wall segments. Along a contour, three consecutive points in space define a circumference, and the smaller the radius of this circumference, the larger the curvature. With these intuitive ideas, the LV endocardial curvature is defined as the inverse of the radius of the circle that can be defined by triplets of adjacent points of the contour delineated on the endocardium. We consider it being concave if the adjacent circle has its centre outside of the left ventricular cavity and convex otherwise see Fig. a. The cu−1) , the LV endocardial contour may contain both convex and concave curvatures, and to distinguish between the two, we consider the curvature convex if the calculated value is positive and concave otherwise.The curvature indexes are calculated for all contour points using a semi-automatic pipeline see Fig. . In a 4CIn order to measure the curvature of the septal wall, the two-dimensional (2D) LV endocardial contour of the end-diastolic frame (used in the diagnosis of BSH ) is requUsing GE EchoPAC strain module, the operator sets a number of points on the endocardium of the LV in the end-systolic phase. Then, the algorithm interpolates the given set, to create additional points, which resemble a smooth curve. The operator then adjusts the segmentation shape by re-positioning the original points. In the next step, the contours are propagated thorough the entire cardiac cycle with the speckle-tracking algorithm.The contours are exported from the module for post-processing as further interpolation is required to avoid underestimating the curvature in critical points. A radial basis function interpolator is chosen, with the smoothness parameter set to the original number of points in the trace as this algorithm provides smooth first and second spatial derivative of the interpolated contour (required in Eq. 1). As a result, each contour is up-sampled to 500 points, a resolution that was empirically tested to lead to robust and smooth curvature values (details on the implementation can be found in ). The enTo test the variability of measurements, an inter-observer/intra-observer analysis was conducted on a group of 20 patients: 10 healthy controls and 10 hypertensive patients, including two cases with BSH. The measurements were performed by clinical experts. The metrics were compared on the basis of the average absolute difference between the measurements relative to the range of measurements, to account for different metric units.First, to evaluate intra-observer variability, the original observer (O1) remeasured the basal and mid-wall IVS dimensions in end-diastole and performed segmentation on all 20 patients at least 2 months after the original measurement (O1∗). Second, to evaluate inter-observer variability, a different observer (O2) measured the same patients using the same 4CH and PLAX images that O1 used. Finally, another observer (O3) measured the same patients using a different 4CH image from the one used by O1 and O2. For most patients, only a single high-quality PLAX image was available, so the IVS was measured in the same PLAX image by O3 as O2 and O1. Similarly, the semi-automatic 4CH segmentation was performed by two other observers on the same (O2) and different (O3) images. This last study gives a measure of inter-observer inter-image variability on 4CH images, which is an important measure of robustness as an ideal method should give consistent results across any image from a patient given reasonable image quality.E/A), and mitral annulus septal and lateral velocities. Moreover, the metrics were studied against the anatomical markers of BSH, namely LV mass, and end-diastolic and end-systolic volumes indexed to BSA. The ability of both metrics, thickness and curvature, to infer the functional impairment is assessed by the regression between the variables within the hypertensive group (N = 161). The agreement between WTR (measured in two views) and ASC to identify BSH among the hypertensive population is studied by their correlation.To capture the deterioration caused by hypertension, functional metrics proven to be affected by BSH  were stut-test is employed to test hypotheses on normally distributed samples, otherwise the Mann--Whitney U test is used. To calculate the correlation, Spearman's ρ and Pearson's Rs are computed.The processing pipeline, the statistical analysis and the variability analysis are implemented using the Python programming language, v.3.6.5. To test for normality of the distribution, the Pearson-D’Agostino test is used . Two sam−1, and 75% of the cases have ASC below −0.88 dm−1. Conversely, WTR distributions are positively skewed, and the 85th percentile denotes the threshold for the diagnosis of BSH with respect to the WTR in either view.As ASC is a novel metric, the distribution analysis is provided as a reference. The distributions of the three metrics under study, the ASC and WTR in 4CH and PLAX, in both the 161 hypertensive and 59 normotensive patients are depicted in Fig. −1, 78% cases with convex curvature index). This agrees with the understanding that the concave ('sigmoid’) septum is a sign of a specific pattern of remodelling [In contrast to the hypertensive population, the distribution of the curvature among the healthy controls was mainly convex .The intra-observer and inter-observer differences relative to the range of measurements were significantly lower for the curvature index when compared with WTR .Assuming current clinical standard the ground truth to identify the presence of BSH in hypertensive patients, we studied how each of the three individual metrics agreed with it. Given that WTR was used as the traditional metric to diagnose BSH, it was not surprising to find a better separation between groups with the WTR indexes than with ASC . ASC was found to also be moderately correlated with the WTR in 4CH view  and weakly correlated to the WTR in PLAX view . Stronger correlation between WTR4CH and ASC measurements was expected as the curvature indexes were calculated from trace delineated in the 4CH view.A more detailed inspection of the metrics in a scatter plot , which is also the most pronounced marker of BSH. In case of the mid-septal strain and left atrial conduit strain, ASC is the only significantly correlated metric. The function of mitral annulus in lateral location is not significantly correlated with any marker.Although the new method does not classify the cases in the same manner, the relation between the metrics and cardiac deformation revealed that ASC was more correlated with the functional remodelling than with the anatomical markers: Table Average curvature of the septal segment is proposed as an anatomical feature to identify the presence of BSH. It is a more precisely defined and reproducible metric than thickness ratios, and better infers the functional remodelling of the basal septal segment.The problem addressed in this work is the high variability between multiple WTR measurements, as described in the literature ,23 and cASC is more reproducible than WTR for two reasons. First, as it has a precise definition, on the contrary to WTR where the locations of the minimum and maximum thickness measurements are ambiguous. ASC is computed along the basal and mid inferoseptal segments of the left ventricle in a 4CH view, objectively derived as 30% of the LV endocardial contour. The second reason is that ASC only requires segmentation on the bright LV endocardial border: curvature defined on the LV endocardial border removes the source of variability in the delineation of the epicardium, which is usually more challenging and required in WTR.There are still two potential sources of variability in ASC. First, there could be inconsistency in placing the first basal points of the endocardial contour, but its impact should be small because of its rigorous description in guidelines . Second,Beyond reproducibility, and in the absence of outcome metrics and disease onset, this work searched for the BSH metric that correlated with those functional markers that indicate early signs of degradation. Changes in left atrial and ventricular function have been shown to be present in hypertensive patients with BSH -- decreased regional LV systolic contraction was related with impaired LV relaxation, a higher level of indeterminate diastolic dysfunction and LA functional impairment . TherefoPrevious studies have explained the mechanistic cause of BSH by the extra stress that this region holds compared with the rest of the left ventricle when pressure increases ,3,5. Givz scores describing the progress of the BSH in hypertensive patients. With such models and follow-up data, both maladaptive (increasing afterload) and adaptive (response to treatment) remodelling predictions could be made, to ensure appropriate treatment.The future methodological research will focus on the automation of the metric acquisition and unveiling the remodelling patterns, to classify the adaptive and maladaptive responses. It has recently been shown that deep neural networks can accurately segment the left ventricle, myocardium and the left atrium directly from ultrasound B-mode images . EmployiIn conclusion, a novel shape biomarker, the LV endocardial septal curvature, was proposed to detect and quantify the presence of BSH. The distribution of this biomarker in control and hypertensive cohorts was described. It was found to be a more reproducible metric than thickness ratios and it better infers the functional deterioration related to hypertension. The tool to compute it , and theThis project has received funding from the European Union's Horizon 2020 Research and Innovation programme under the Marie Skłodowska-Curie grant agreement No 764738. P.L. holds a Wellcome Trust Senior Research Fellowship (209450/Z/17/Z).There are no conflicts of interest.


Glaucoma is a chronic ocular disease characterized by damage to the optic nerve resulting in progressive and irreversible visual loss. Early detection and timely clinical interventions are critical in improving glaucoma-related outcomes. As a typical and complicated ocular disease, glaucoma detection presents a unique challenge due to its insidious onset and high intra- and interpatient variabilities. Recent studies have demonstrated that robust glaucoma detection systems can be realized with deep learning approaches. The optic disc (OD) is the most commonly studied retinal structure for screening and diagnosing glaucoma. This paper proposes a novel context aware deep learning framework called GD-YNet, for OD segmentation and glaucoma detection. It leverages the potential of aggregated transformations and the simplicity of the YNet architecture in context aware OD segmentation and binary classification for glaucoma detection. Trained with the RIGA and RIMOne-V2 datasets, this model achieves glaucoma detection accuracies of 99.72%, 98.02%, 99.50%, and 99.41% with the ACRIMA, Drishti-gs, REFUGE, and RIMOne-V1 datasets. Further, the proposed model can be extended to a multiclass segmentation and classification model for glaucoma staging and severity assessment. Glaucoma is a common ocular disease caused by high intraocular pressure, which can eventually lead to irreversible blindness. It is a complex disease, influenced by patient's age, race, refractive error, OD, and retinal nerve fiber layer (RNFL) . OD is aSeveral clinical studies have been performed to elucidate the relationship between glaucoma and the OD in terms of its structural changes, including thinning, cupping, excavation, and disc hemorrhage. One such investigation before two decades reported in  describeClassification of glaucomatous disc changes is challenging due the mixed morphological manifestations of hemorrhage patterns on the surface of the OD. Cup to disc ratio (CDR) , evaluatSegmentation of ODs and cups is a nontrivial problem where inaccurate segmentations may result in false detections. In the recent years, deep learning approaches are gaining popularity in the deployment of automated glaucoma detection. Deep learning networks capable of learning image features at various levels of abstraction are employed as feature extractors and classification and segmentation models in glaucoma screening. Several two-stage convolutional neural network- (CNN-) based glaucoma detection frameworks which perform segmentation followed by classification are proposed in –14. In tExisting works in literature pay less attention to the contextual information in glaucoma screening. While the models which segment the OD and OC separately focus on clear separation of boundaries for CDR calculation, the joint segmentation approaches are based on heavy learning models such as generative adversarial networks (GAN) and ensemble-based architectures. This research presents a context aware joint segmentation and classification framework for glaucoma detection based on the aggregated transformations and class activation maps (CAM). The framework called GD-YNet follows a YNet architecture  comprisiA joint segmentation classification framework based on Ynet architecture which demonstrates superior OD segmentation and glaucoma classification performances is proposedA hierarchical approach for RoI segmentation based on class activation maps (CAM) and aggregated transformations is proposedThe prosed framework is designed to harnesses the low-level and structural features of the fundus image in glaucoma detectionThe contributions of this research are as below. The proposed joint model is trained and tested with public databases, demonstrating superior segmentation and classification performances. From the objective metrics and explainable analysis, it is seen that the proposed model is a prospective solution for integration with glaucoma screening pipelines for better clinical outcomes.Rest of this paper is organized as below. In Existing deep learning approaches for glaucoma detection are based on feature extraction, object recognition, transfer learning, RoI segmentation, OD and OC segmentation, and feature discrimination. The earliest deep learning model for glaucoma detection proposed in  is deploLater, several deep learning models for glaucoma detection based on OD and OC segmentation have been proposed. In , the conA dual machine learning system  is builtIn , OD and A depth guided semantic segmentation network is employed in  for joinRetinaGAN  based onThe context encoder network CENet  capturesA spatial-aware neural network (SAN)  based onA U-shaped CNN built with multiscale input and multikernel modules (MSMKU) proposed in  capturesIn , joint sRecurrent fully convolution network (RFC-Net)  is desigA disc-aware ensemble network (DENet)  is modelAn adversarial learning framework called patch-based Output Space Adversarial Learning framework (pOSAL)  performsGLNet  is a mulThough several deep learning frameworks are employed in OD and OC segmentation and glaucoma detection, exclusive studies to evaluate the performance of several deep learning classifiers on diverse datasets do not exist. One such investigation by Sreng et al.  presentsThe DenseNet supports feature propagation between layers with direct connections and facilitates feature reuse. The Fully Connected DenseNet (FC-DenseNet) model  for simuDespite several deep learning models deployed in glaucoma detection, surprisingly, new models following different learning paradigms have been proposed in 2020. A framework proposed in  for OD sThe fuzzy broad learning system (FBLS)  for glauThree representative deep learning models for glaucoma detection are proposed in 2021, each following an unique architecture. A two-stage glaucoma detection model proposed in  employs An attention UNet  construcA convolutional autoencoder- (CAE-) based segmentation network for OD segmentation is proposed in . InitialFrom a thorough review of automated glaucoma detection literature, it is evident that context aware segmentation approaches yield best classification results in glaucoma screening, rather than conventional classifiers based on extraction of deep features from the fundus images. Recently, joint segmentation methods are getting wide research attention due to the simplicity of the design. However, their implementation is complex due to the intricacies involved in deploying them as multilabel segmentation approaches. Further, there are few works on multitask learning such as , 25 whicThis section presents the training and testing datasets employed in this research and a succinct description of the underlying methods.The public datasets, viz., ACRIMA, Drishti-gs, REFUGE, RIGA, and RIM-ONE are employed in training and testing the proposed system. The unified segmentation and classification model proposed in this research is trained and tested with discrete datasets without any data augmentation, to evaluate its generalization ability. The details of the datasets are given in The proposed framework is implemented with Matlab 2021a software, employing the image processing and deep learning toolboxes. The model is trained and tested with a NVIDIA GeForce GTX1060 3GB Graphics card enabled i7-7700 K processor with 32GB DDR4 RAM.Training deep networks is challenging due to the vanishing gradient and gradient explosion issues. To solve these problems, the residual learning approach was proposed in . In thisThe ResNeXt architecture  is an exThe CAMs generated from the final convolutional layer of a CNN provide a visual representation of the behavior of the classifier model, highlighting the areas of the image which are the most important for the classification. These maps are generated by first pooling the features of the neurons in the last layer of the CNN and then passing this information through a softmax function, to produce a probability distribution over the classes of the image. In this way, the activations of the final layer of the CNN can be viewed as a feature map, which can be interpreted as a visual representation of the model's classification decision, highlighting the topological and spatial distribution of its decision boundaries. The activation maps can be used in debugging the model to generate class-specific features for training a classifier.However, CAMs are limited to classification problems, and using these coarse maps for localization is still a challenge. Jiang et al.  proposedi, j) in a feature map f in a layer l is given in Equation (A(∙) is the activation function. In this research, the LayerCAM is constructed from the image gradients as described below. The weight of a spatial location (fand corresponding weights as in Equation (The activation of the entire feature map Equation . The LayEquation . (2)Fi,jThe schematic of LayerCAM construction is shown in The proposed joint segmentation and classification model for OD segmentation and glaucoma detection is realized with a YNet architecture originally proposed in . YNet faGenerally, vanilla CNNs such as EfficientNet, Inception, ResNet, and VGG are employed as the backbones of the encoders and decoders defining the organization of the layers in the network. CNNs for image processing problems use fixed sized filters and exercising the networks to find optimal filter sizes requires an intensive tuning procedure, that is not easy to automate. Further, fixed size filters are suitable only for images with similar size salient parts, and they fail to adapt to images with varying scale. Inception networks employ filters of multiple sizes and shapes together in a single network. Such networks are shallow but can operate in a wide range of scales. The inception network is a deep convolution neural network that can be trained to detect salient regions in any image, and it can be used to generate new images that contain the same salient elements. Inception networks contain layers with multiple sized and shaped filters. For each layer, a region-of-interest (ROI) is extracted, and a convolution is computed on the input image with the filters that are applied to the ROI. The results of these computations are concatenated with the results from the previous layer. This facilitates the network to learn to identify features of different sizes. In this research, YNet is constructed from UNet which is built on the Inception blocks. The structure of the inception block is illustrated in OD morphology is a powerful predictor of the outcome of medical and surgical treatment and is one of the most significant prognostic factors in the management of patients with chronic glaucoma. Heterogeneity of OD provides insight into the pathophysiology of glaucoma and is associated with disease progression. The proposed framework is realized in two stages, viz., RoI segmentation followed by joint OD segmentation and glaucoma detection as shown in RoI segmentation is realized by constructing the LayerCAM from the feature maps of the ResNext subnetwork. Segmentation of OD and glaucoma detection is performed with the joint segmentation and classification subnetwork, implemented with the inception YNet network, unifying inception blocks and the YNet architecture described in the previous section. The schematic of this subnetwork is shown in The basic Ynet proposed in  is only At the lowest layer of the network, the feature map from the encoder path is subjected to a set of four convolutions and shared with the inception module in the decoder path. The resulting feature map is upsampled, doubling the width and height and concatenated with the feature map generated in the encoding path at the same layer. This results in a subsequent increase in depth of the feature maps by 8. Finally, the feature map generated at the top most layer of the decoder path matches the dimensions of the input image, and it is convolved with a set of 1 × 1 convolutions, by which the depth of the feature map is reduced to match the number of class labels. A binary segmentation mask is constructed from this feature map applying a pixel-wise activation function. The OD is obtained by masking it with the fundus image, and it is downsampled by average pooling. The low-level feature map from the last encoder stage is upsampled to match the size of the OD and concatenated with it. The resultant image of size 256 × 256 is fed as input to a convolutional layer with two 3 × 3 filter to construct a feature map of size 128 × 128. This is followed by two fully connected layers with sigmoid activation functions. The first fully connected layer generates a feature vector of size 64 × 1, and the second one generates the class scores.The inception modules consists of 1 × 1, 3 × 3, and 5 × 5 convolutional filters and max pooling layers, generating four feature maps from the input and concatenating them. The 3 × 3 and 5 × 5 filtering operations are preceded by 1 × 1 convolutions to reduce the number of channels, and the maxpooling layer is used to create an abstract representation of the feature map before 1 × 1 convolution. The inception module is designed on the premise that salient parts of medical images are of arbitrary sizes and are often localized in a small spatial region. Further, the features should be invariant under changes in the input size to maintain invariance between different scales of objects. The Inception modules with different filter sizes integrated with the UNet architecture can capture multiple scales of optic disc features with various orientations, which can be learned efficiently to discern the OD. It is to be noted that the size of the input image, the number of classes, and the number of inception modules are all parameters of the proposed framework. Further, training this network with the cropped or resized candidate RoIs extracted from the Layer CAMs can further simplify the segmentation process.The proposed system constitutes subnetworks for RoI segmentation, OD segmentation, and classification. The training parameters of the framework are listed in For RoI segmentation, as seen in Lseg and classification Lcls as as described below. The network is optimized by alternating minimization of the segmentation loss and the classification loss in each iteration, thus improving the performance of both tasks. The objective function of the model is expressed in Equation (The objective function of the joint segmentation and classification framework is expressed as a combination of segmentation loss Equation . (4)minΘN is the training set size, and yi is the label of the segmentation mask for the sample i . The segmentation loss can be formulated as in Lsegyi denotes the binary crossentropy loss with yi, Fθ(xi) is the predicted segmentation mask, and σ is the loss weight in the range [0 1]. The classification loss can be formulated as in (where 1 − ed as in . (6)LclsLseg penalizes the deviation from the ground truth segmentation mask, while Lcls encourages the segmentation to be as close to one as possible. yi can be 0 or 1 depending on the ground truth and network predictions.In this minimization problem, This section presents the OD segmentation and glaucoma detection results on the datasets. Initially, the RoI is localized with the LayerCAM constructed from each training image to capture the candidate ODs. The activations at different channels of a convolutional layer, and the LayerCAM construction are shown in Figures The proposed inception-based YNet framework is initially fine-tuned for segmentation with the MESSIDOR dataset, and it is tested with the RIMOne-V2 dataset. The segmentation results for a subset of images are shown in The segmentation and classification performances are evaluated with the accuracy, sensitivity, specificity, precision, DC, Jaccard coefficient (JC), structure measure (SM), and enhance-alignment measure (EM) metrics. The ground truths required for these evaluations are provided with the RIMOne-V2 dataset.Accuracy is a measure of the number of correct predictions out of the total number of samples, which is derived from the true positive (TP), true negative (TN), false positive (FP), and false negative (FN) values as in . (7)AccuWhile sensitivity and specificity refer to the proportions of true positives and true negatives correctly identified, respectively, in OD segmentation problem, these metrics refer to the number of correctly identified pixels of the OD and the background, respectively, as in Equations  and 9)..9). (8)SPrecision refers to the number of correctly identified samples of the total number of predicted positive samples as in Equation . (10)PreP and the ground truth G, is given in Equation DicJC also called intersection-union (I-U) measures the similarity and difference between the prediction and the ground truth. So, Sr, α, Sp and G refer to the object-aware similarity, region-aware similarity, trade-off between object and region awareness, predicted mask, and the ground truth, respectively, where α = 0.5 by default. The SM which evaluates the similarity between the segmented mask and the ground truth based on object and region awareness is given in Equation , where Sw and h are the width and height of ground truth mask G, and φ is the enhanced alignment matrix. EM is a measure of the global and local similarity between binary maps of the segmented and ground truth masks as given in Equation , where wSp, EMφ is obtained from a binary mask, thresholding each pixel in the range [0 255]. The alignment matrix φ captures the similarities between the predicted mask and the ground truth at the pixel and image levels, from the global means. All these metrics range from 0 to 1 and evaluate to a value closer to 1 when the predicted segmentation masks are similar to the ground truth. Segmentation metrics for the Drishti-gs dataset is compared with that of state-of-the-art models in Given a prediction From The receiver operating characteristics (ROC) of the classifier model for the four datasets are shown in Further, a comparison of the classification results with representative works is presented in Recently, explainable artificial intelligence (XAI) has emerged as one of the key methodologies to enable trust, acceptance, and adoption of machine learning models in several domains. XAI complements the learning models by providing transparent explanations for the decisions based on the data. Generally, XAI is performed with CAMs and gradient CAMs constructed from the final learnable layer of a deep learning model. In this paper, the LayerCAM construction for RoI segmentation is based on multiple CAMs as described in It is observed that the class scores for the correct classifications are closer 1 while they are comparatively lower for misclassifications. It is seen the GCAM for the misclassified normal image is very small, and that of the glaucomatous image is not intact. The GCAMs show that the ODs are of diverse morphologies for normal and glaucomatous images. A detailed analysis of the CAMs can facilitate the definition of the OD morphologies in the staging and severity analysis of glaucoma.Experimental results reveal that the proposed model demonstrates superior performance with few misclassifications on all test datasets. This shows that the experimental setup is ideal for the glaucoma detection problem. In this scenario, it is unrealistic to study the performance of the model by increasing the depth of the YNet architecture. Since the classification accuracy depends on the segmentation of the ODs, it is appropriate to perform ablation study with the ResNeXt network used in LayerCAM construction.As shown in The efficacy of the proposed glaucoma detection model is established with the experimental results, comparative analyses, and explainable studies. Further, XAI analysis shows that OD morphologies of arbitrary shapes and sizes are captured by the proposed model. Superior performance of the proposed framework is attributed to the context aware RoI segmentation using LayerCAM. The ResNeXt model, commonly used as a pretrained classifier or encoder backbone in semantic segmentation networks, is employed as a feature extractor in the construction of LayerCAM. Several existing models initially perform cropping for RoI or OD segmentation. However, several images in the dataset captured at different orientations do not have the ODs oriented to the image centers, and cropping them results in loss of the RoI. To the contrary, the proposed model localizes the RoIs from the LayerCAMs constructed from the ResNeXt blocks aggregating the finer and coarse details of the OD.The proposed GD-YNet resembles the joint segmentation and classification architectures of , 25. TheThe Ynet proposed in  for joinThough the architecture of the proposed GD-YNet matches the schematic of the YNet, the proposed framework is superior to the works in , 25 in tIn spite of the demonstrated efficacies of the proposed framework in segmentation and classification, certain limitations have been identified which must be addressed in future. The first limitation is concerned with the architecture of the ResNeXt blocks. The cardinality value of the ResNeXt blocks is arbitrarily assumed as 32. LayerCAM with different cardinalities and group convolutions can be used to study the performance of the model with respect to these parameters towards optimization of the cardinality and filter sizes. The second limitation is the usage of the basic inception modules with dimension reduction abilities in the encoder-decoder paths. Recently, variants of inception networks have been introduced with factorized 7 × 7 convolutions, root mean square (RMS) optimizers, and reduction blocks. The naïve inception blocks may be replaced by these variants for improved performances.This paper presents a novel framework based on YNet architecture for joint segmentation and classification in glaucoma detection. It is implemented in two stages with context aware RoI segmentation, leveraging the potential of the aggregated transformations of the ResNeXt architecture in the first stage and a light-weight classification branch in the second stage. The proposed GD-YNet model based on this framework is trained to segment the ODs from the segmented RoIs followed by binary classification from the ODs. Tested with public datasets, this framework demonstrates superior segmentation and classification performances compared to state-of-the-art models. Chronic glaucoma manifests diverse morphological patterns in the ODs such as disc hemorrhage, nerve fiber layer defects, nerve fiber layer swelling, and atrophy. The proposed framework can be extended to multiclass segmentation and classification in the detection of pathologies pertaining to neuroretinal rim losses, disc cupping, rim thinning, disc hemorrhage, etc. This requires extensive training of the proposed model with several classes to differentiate morphological patterns of the ODs.


This paper proposes a new generative adversarial network for infrared and visible image fusion based on semantic segmentation (SSGAN), which can consider not only the low-level features of infrared and visible images, but also the high-level semantic information. Source images can be divided into foregrounds and backgrounds by semantic masks. The generator with a dual-encoder-single-decoder framework is used to extract the feature of foregrounds and backgrounds by different encoder paths. Moreover, the discriminator’s input image is designed based on semantic segmentation, which is obtained by combining the foregrounds of the infrared images with the backgrounds of the visible images. Consequently, the prominence of thermal targets in the infrared images and texture details in the visible images can be preserved in the fused images simultaneously. Qualitative and quantitative experiments on publicly available datasets demonstrate that the proposed approach can significantly outperform the state-of-the-art methods. Image fusion is a significant technology to enhance images, aiming to synthesize images by integrating complementary information from several source images captured by different sensors ,2. It haOver the past few years, increasingly, infrared and visible image fusion methods have been proposed. The methods can be classified as multi-scale transform-, sparse representation-, neural network-, subspace-, saliency-, and deep learning-based methods and hybrid methods. The existing fusion methods have three key points: image transformation, activity level measurement, and fusion rule designing. Based on these three points, a large number of successful fusion methods have emerged.Although existing fusion methods can achieve good results, several aspects can be improved. First, artificial rule designing makes the hand-crafted methods more and more complex and time consuming . Second,To address these challenges, a generative adversarial network is proposed for infrared and visible image fusion based on semantic segmentation (SSGAN). Firstly, the proposed method is a deep learning-based method without artificial rule designing, which is an end-to-end fusion network with high efficiency. Secondly, the semantic information of the source image is taken into account. The source images are divided into the foregrounds and the backgrounds by utilizing the masks obtained by semantic segmentation. The foregrounds represent the salient semantic object, while the backgrounds represent the target with finer texture details. Thirdly, the generator is divided into the foreground- and the background-path to extract the corresponding information in source images, respectively. Finally, to overcome the lack of a ground-truth, the input images of the discriminator are designed by combining the foregrounds of the infrared images with the backgrounds of the visible images. This can promote the fusion images to preserve more texture details and the thermal radiation information concurrently. Qualitative and quantitative results reveal the advantages of the proposed SSGAN compared to other methods.The contributions of the current work include the following aspects:First of all, in order to extract the features of source images better, semantic segmentation is introduced into fusion tasks to divide the source images into the foregrounds and backgrounds. It is convenient to adopt a more targeted feature extraction strategy for the foregrounds and backgrounds.Secondly, to extract various semantic information, a generator is designed with a dual-encoder-single-decoder model, encoding the corresponding features for different semantic regions by the dual-encoder and fusing different semantic features by the decoder.Then, the image obtained by combining the foregrounds of the infrared images with the backgrounds of the visible ones is applied as the reference data to make the fused images retain abundant information.https://github.com/Jilei-Hou/FusionDataset, accessed on 20 March 2021.Finally, in order to get the semantic information, the infrared and visible image fusion dataset is annotated, including 60 typical infrared and visible image pairs with labeled images. The dataset is available at The rest of the paper is organized as follows. In this section, the infrared and visible image fusion approaches, the recent research work related to semantic segmentation, as well as GANs and their variants are reviewed.Numerous infrared and visible image fusion methods have been proposed in recent years, which can be divided into seven categories, including multi-scale transform-based methods, sparse representation-based methods, neural network-based methods, subspace-based methods, saliency-based methods, deep learning-based methods, and hybrid methods.Multi-scale transform-based methods ,13 are tSparse representation-based methods ,17 have Neural network-based methods have been popular ways of performing image fusion tasks over the past few years , due to Subspace-based methods ,22 aim tSaliency-based methods ,24 imitaRecently, since the rise of deep learning, a large number of deep learning-based fusion methods have been proposed ,27. Li eThe above-mentioned infrared and visible image fusion methods all have their advantages and disadvantages. Hybrid methods ,31,32 caSemantic segmentation is a typical topic in computer vision that involves taking raw data  as the input and converting them into masks with highlighted areas of interest. Unlike other topics that only focus on the edge, semantic segmentation fully provides pixel-level image understanding in a human-perceived way by bringing together parts of an image that belong to the same target. Compared to other image-based tasks, semantic segmentation involves high-level cognition, which achieves the identification of the content and the corresponding location in the image.Long et al.  proposedG and a discriminative model D. The generator can generate realistic images from random noise. The discriminator is trained to distinguish the generated data from the real data. It can make the generator learn the data distribution through a continuous game between the generative model G and discriminative model D. Compared with the traditional model, it has two different networks, not a single network, and the training method is adversarial training. The gradient update information of G in GAN comes from the discriminator D, not from data samples. The adversarial training between the generator and discriminator is continued until the discriminator cannot distinguish the generated samples. Subsequently, a relatively more realistic image sample can be produced by the generator.Goodfellow et al.  first prThe success that the original GAN has achieved is limited due to the unstable training process. To solve this problem, CGAN  extendedGANs have achieved satisfactory results in infrared and visible image fusion. FusionGAN , proposeIn this section, the proposed SSGAN for infrared and visible image fusion is presented. We begin with a formulation of the infrared and visible image fusion problem as image generation with GAN, and then, the loss function is discussed. Finally, the network architectures of the generator and the discriminator are described.Infrared images can distinguish targets from their backgrounds due to the difference in thermal radiation even in poor lighting conditions. Visible images can represent texture details with a high spatial resolution. Meanwhile, both preserve the corresponding semantic information. Therefore, infrared and visible image fusion should keep both the radiation information of the infrared images and the texture details of the visible images. The fused images also need to reserve the high-level semantic information of both source images. Additionally, for different semantic targets, the information that fusion images want to retain from infrared and visible images is different. Specifically, the semantic object with thermal radiation information in infrared images takes precedence over the object’s information in visible images. In contrast, the semantic object with texture details in visible images takes precedence over the object’s information in infrared images.In order to take semantic information into account in the fusion task, this paper proposes a generative adversarial network based on semantic segmentation, named SSGAN. According to the difference of thermal radiation information and texture details, each source image can be divided into a foreground and a background by the mask. The foreground is mainly composed of semantic targets that contain rich thermal radiation information in the infrared image, such as a person and a car. The background is mainly made up of semantic objects with rich texture details in the visible image. The proposed SSGAN can use different feature extraction methods for the foreground and background to better retain the information of the source image. The procedure of the proposed SSGAN is shown in G, whose output is a fused image D is employed to establish an adversarial game with the generator. Specifically, the generator aims to generate an image that can fool the discriminator. The discriminator is committed to distinguishing the generated image from the real image. As the adversarial game proceeds, a better fusion image can be generated by the generator. Moreover, in order to make the discriminator work better, we redesign the discriminator’s input based on semantic segmentation. This is obtained by blending the foreground of the infrared image and the background of the visible image. The production process of the real image In the beginning, the mask with the infrared image’s semantic information is obtained by deeplabv3+, which is a successful model used for semantic segmentation. Then, we take the infrared image Equation .(1)Id=IG conditioned on a visible image G, is encouraged to be realistic and informative enough to fool the discriminator. By establishing an adversarial game between the generator G and discriminator D, the fused image G and discriminator D can be formulated as in Equation minGG consists of three parts: adversarial loss G, G and discriminator D, which is defined as in Equation LG=λEquation .(4)LadvH and W represent the height and width of the source images, respectively. The second item Equation .(5)LconIt is worth noting that the texture details of visible images and the thermal radiation information of infrared images should be mainly preserved in the fused results, while the thermal radiation information of visible images and the texture detail of infrared images are of secondary importance. Therefore, x and y can be defined as in Equation (The last term Equation .(6)LSSIEquation  45].LD=1As shown in For each encoder path, four layers with 3 × 3 filters are adopted for feature extraction. Meanwhile, inspired by DenseNet , the denThe specific settings of all layers are shown in D of the proposed SSGAN is a simple and effective five-layer convolution neural network with well-designed real data. The discriminator architecture is shown in G. The stride of all convolutional layers was set as two.The discriminator plays an adversarial role against the generator to make the generator’s fused image more realistic and visual. Unlike the traditional single discriminator, it takes only one source image as real data or the dual-discriminator takes the infrared image and visible image as real data at the same time. The discriminator https://figshare.com/articles/TNOImageFusionDataset/1008029, accessed on 20 March 2021) and the INO dataset . First, the experimental datasets and training details are introduced. Second, the parameters used in the proposed method are discussed. Third, a series of recent state-of-the-art methods for comparison and commonly used evaluation indicators are described in detail. Then, the experimental results on the TNO dataset and INO dataset are analyzed. Next, the time complexity is also considered. Finally, the ablation experiments related to the mask and discriminator are illustrated.This section focus on verifying the effectiveness of the proposed algorithm through a large number of qualitative and quantitative experiments on the TNO dataset , the standard deviation (SD), mutual information (MI), and visual information fidelity (VIF).L denotes the gray level of the image and i in the fused image.Entropy (EN) is a common method for statistical image features, reflecting the amount of information obtained from infrared and visible images. The mathematical definition of entropy is defined as in Equation  52]. represents the dispersion of image gray value relative to the average gray value. The definition of the standard deviation can be defined as in Equation  53]. is a basic concept in information theory, which can measure the correlation between two random variables. In image fusion, mutual information is used to measure the correlation between the source images and the fused images. The definition of mutual information in infrared and visible image fusion is defined as in Equation  54].(1.(154].Equation .(12)MIXThe visual information fidelity (VIF)  metric mQualitative comparisons can evaluate the quality of the fusion image based on the human visual system. Since visible images conform to human visual habits, the fused results of the infrared and visible images should conform to human visual habits to a certain extent. To give some intuitive results on the fusion performance, we selected four typical image pairs from the TNO dataset for qualitative evaluation. The fusion results of the proposed SSGAN and the other five comparison methods are shown in The first two columns in However, different fusion methods have their own unique performance, and the infrared thermal target is very obvious in the fused results of GTF and FusionGAN. In other words, the thermal radiation information of the infrared images is preserved successfully. On the other hand, although the salient target information in the fused results obtained by GTF and FusionGAN is well preserved, some of the texture details in visible images are missing. In contrast, for other comparison methods like DTCWT, wavelet, and DenseFuse, the targets  in the fused images are not as obvious as those in the infrared images. This means that the thermal radiation information in the infrared images is not well preserved, but their texture details are well preserved.In general, all the comparison methods exploit the information in the source images, either thermal radiation information or texture details. Different from all the comparison methods, the proposed SSGAN is devoted to preserving both thermal radiation information and texture details at the same time. For example, in the second column, the fused images of SSGAN highlight people while retaining the texture of the ground. A similar phenomenon can also be observed in other examples. This demonstrates that the proposed SSGAN has better performance than the other state-of-the-art methods in terms of simultaneously preserving thermal radiation information and texture detail information.The proposed SSGAN was further compared with the five state-of-the-art methods quantitatively on 26 image pairs. The results are summarized in The proposed method and the other five comparison methods were further evaluated on the INO dataset, which is also a dataset commonly used for infrared and visible image fusion. Sixteen visible and infrared image pairs were selected from the video named The qualitative fused results of the proposed SSGAN and the other five comparison methods are shown in In addition to qualitative experiments, the proposed SSGAN was further compared with the above-mentioned competitors quantitatively on 16 image pairs from the INO dataset with four metrics, i.e., entropy (EN), standard deviation (SD), mutual information (MI), and visual information fidelity (VIF). The results are summarized in The runtime comparison of six methods is also provided in In order to generate a high quality fused image and overcome the shortcoming of lacking of a ground-truth, we designed the input image The comparison results are given in To verify the effect of the game between the generator and discriminator in GAN, we removed the discriminator, and the whole network merely contained the generator. Thus, the adversarial relationship no longer existed. The training goal was to minimize the content loss This paper presents a new approach for infrared and visible image fusion. The key characteristic of the proposed method is that it applies semantic segmentation to image fusion, and the network structure adopts a dual-encoder-single-decoder to extract features. The proposed method uses the powerful GAN technique, and the reference data of the discriminator are redesigned based on semantic segmentation. The experiments on the TNO and INO datasets demonstrate that the proposed approach yields results that consistently outperform the state-of-the-art methods, both qualitatively and quantitatively.However, the current work verifies the effectiveness of semantic segmentation for the fusion task, and the segmentation network is trained separately. In the future, we will try to use a unified network for fusion and segmentation and build segmentation consistency loss between the fused image and source images. The segmentation consistency loss can constrain the semantic consistency between the fused image and the source images, which will enforce the network to generate a fused image with clearer semantic targets and richer texture details.