Approximately 12.3% of mothers experience childbirth-related post-traumatic stress symptoms (CB-PTSS). However, evidence-based interventions to treat CB-PTSS are lacking. Intrusive memories (IM), a key CB-PTSS, are distressing and can trigger other PTSS by reliving the traumatic event. Emerging evidence shows that a behavioural intervention involving a visuospatial task (BI-VT) can reduce the number of IM and PTSS, supposedly by interfering with the reconsolidation of the trauma memory. This study aims to test the efficacy of a single-session BI-VT targeting IM to reduce the number of childbirth-related (CB-)IM and PTSS, in comparison to a waitlist control group (WCG).In this multicentre, single-blind, randomised controlled trial being undertaken at one regional and one university hospital in Switzerland, 60 participants will be allocated to the Immediate Intervention Group (IIG), receiving the immediate intervention on day 15, and 60 participants to the WCG receiving the delayed intervention on day 30. All participants will report their CB-IM during the 2 weeks preimmediate and postimmediate intervention in diaries. The IIG will additionally report their CB-IM over weeks 5 and 6 postimmediate intervention. Self-report questionnaires will assess CB-PTSS at 2 weeks preimmediate and postimmediate intervention in both groups, and at 6 weeks postimmediate intervention in the IIG. A feedback questionnaire will evaluate the intervention acceptability. The primary outcome will be group differences in the number of CB-IM between the 2 weeks preimmediate and postimmediate intervention. Secondary outcomes will be CB-PTSS at 2 and 6 weeks postimmediate intervention, the number of CB-IM at weeks 5 and 6 postimmediate intervention, and intervention acceptability.Ethical approval was granted by the Human Research Ethics Committee of the Canton of Vaud (study number 202200652). Participants will provide an informed consent before study participation. Results will be presented in peer-reviewed journals and at conferences.NCT05381155. This study aims to contribute to the development of an innovative, single-session and easily accessible intervention to reduce childbirth-related post-traumatic stress symptoms (CB-PTSS).A pilot study has already shown high efficacy and acceptability of this single-session intervention.CB-PTSS will be assessed with a validated questionnaire specifically developed for the perinatal context.Participants will be blinded to group allocation, but the research team will not.The waitlist control group is not an active comparator.Approximately 12.3% of mothers develop childbirth-related post-traumatic stress symptoms (CB-PTSS) and 4.7% of mothers meet the diagnostic criteria for childbirth-related post-traumatic stress disorder (CB-PTSD).For such interventions, IM are targets of particular interest since they involve re-experiencing distressing moments of the trauma and may trigger or maintain other PTSS.Each new event initiates memory consolidation, a time-dependent process during which memories are malleable until their stabilisation into long-term memory within hours.Although emerging evidence supports the efficacy of interventions assumed to interfere with trauma memory consolidation to prevent the development of PTSS,Interventions assumed to interfere with memory (re)consolidation have used various interference strategies, such as pharmacological agents, psychological therapy techniques or behavioural interventions involving a visuospatial task (BI-VT).26A single-session BI-VT was recently adapted to the perinatal context in a pre–post pilot study (n = 18).efficacious to reduce CB-PTSS.These preliminary results suggest that a single-session BI-VT is suitable for the perinatal context. Building on this pilot study, a randomised controlled trial (RCT) will then allow to control for other potential confounding factors, such as the time between the trauma and the intervention.One challenge for the translational research on memory reconsolidation is to identify if each supposed memory process is taking place and at which point.38This study aims to test the efficacy of a single-session BI-VT in reducing the number of CB-IM and CB-PTSS, in comparison to a waitlist. This intervention is assumed to reactivate the childbirth memory and interfere with its reconsolidation, respectively through a brief childbirth narration and Tetris gameplay, leading to a decrease in CB-IM and CB-PTSS. The primary outcome will be the difference in the number of CB-IM between baseline (2 weeks preimmediate intervention) and the 2 weeks postimmediate intervention in the Immediate Intervention Group (IIG) compared with the WCG .Mothers with a minimum age of 18 years, who gave birth at least 6 weeks prior to the screening in one of the two study centres , and suffer from CB-IM will be eligible to participate. To avoid a floor effect regarding CB-IM reduction, we will only include mothers reporting at least four CB-IM over the 2 weeks prior to the screening. Exclusion criteria will be insufficient French-speaking level to complete the assessments, established intellectual disability or psychotic illness, an ongoing psychological treatment in relation to childbirth, propranolol medication  will screen interested mothers. The presence and number of CB-IM over the past 2 weeks will be assessed with the item B1 of the Clinician-administered PTSD scale for Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, (DSM-5),All participants will receive questionnaires regarding sociodemographic and obstetric information, postnatal depression and anxiety symptoms on the first day of the baseline. They will report their CB-IM in diaries during the 2 weeks preimmediate intervention. They will also report their CB-PTSS in a self-report questionnaire during week 2.The IIG will receive the intervention on day 15. For a description of intervention procedures, see the Intervention section.Both groups will report their CB-IM during the 2 weeks postimmediate intervention. To ensure blinding of WCG, this group will complete a diary covering T1 and T3. Both groups will report their CB-PTSS in a self-report questionnaire during week 2 postimmediate intervention.The WCG will receive the delayed intervention on day 30. The delayed intervention follows the exact same procedures as the immediate intervention.The WCG will report their CB-IM in a diary during the 2 weeks postdelayed intervention and their CB-PTSS in a self-report questionnaire during week 2 postdelayed intervention.The IIG will report their CB-IM in a diary during weeks 5 and 6 postimmediate intervention. They will also complete self-report questionnaire assessing postnatal depression and anxiety symptoms, as well as CB-PTSS during week 6 postimmediate intervention. The WCG will not complete follow-up assessments to ensure a fair distribution of burden between groupsBoth groups will complete a feedback questionnaire by phone.The study coordinator, a trained psychologist, will meet the participant in a neutral room, inside a non-medical administrative building, explain intervention procedures, and attach an ECG device to their chest. To assess HRV baseline, the participant will count aloud during 2 min . The study coordinator and the participant will then go into a gynaecological examination room of the maternity ward to realise the first part of the intervention:1. Childbirth narrative task (assumed memory reactivation phase): the participant will be asked to briefly narrate their childbirth orally during 10 min. The study coordinator will first ask participants to narrate the overall course of their childbirth during 5–7 min (Narrative task A). Then, the participant will be asked to narrate the moment associated to the most frequent or distressing CB-IM reported in the diary, for another 3–5 min (Narrative task B). If the participant perceived her distress to be overwhelming, she will have the option to go into another room not located in the maternity or to write about her childbirth instead of narrating it orally.The participant and the study coordinator will then return to the neutral room to start the second part of the intervention, at least 10 min later, in order to allow the assumed memory labilisation2. Tetris gameplay (assumed memory reconsolidation disruption phase): the study coordinator will ask the participant to play Tetris on a handheld gaming device (Nintendo 3DS), in ‘Marathon’ mode, with both sound and 3D switched off. The participant will be instructed to focus both on the block currently falling and on the upcoming blocks (shown in a preview box). To promote mental rotation, participants will be encouraged to use their imagination to visualise how to place each block to complete as many horizontal lines as possible.After Tetris gameplay, the participant will count aloud for 2 min to measure HRV recovery . Finally, the participant will be asked not to play Tetris or to research the therapeutic use of Tetris until she terminated the study.An overview of the intervention procedures is shown in The primary outcome will be the difference in the number of CB-IM between baseline and the 2 weeks postimmediate intervention across groups.Secondary outcomes will be:The change in CB-PTSS between: (a) baseline and at 2 weeks postimmediate intervention across groups, (b) baseline and follow-up in the IIG.The change in the number of CB-IM between baseline and follow-up in the IIG.The acceptability of the intervention in both groups.This study will investigate the following exploratory outcomes in both groups:The course of the distress, nowness, sensorial modalities, and content related to CB-IM before and after the intervention.The association of HRV and subjective distress assessed during the different phases of the intervention, with the intervention efficacy (number of CB-IM and CB-PTSS postintervention).In addition, exploratory outcomes will include the following in the IIG:The change in postnatal depression, and anxiety symptoms between baseline and follow-up.The change in the number of CB-IM between the 2 weeks postimmediate intervention and follow-up.The change in CB-PTSS between 2 weeks postimmediate intervention and follow-up.We will also explore, in the WCG, the change in the number of CB-IM and CB-PTSS between the 2 weeks predelayed and postdelayed intervention.not at all to 10 = extremely: (1) how distressing it was, (2) to what extent they had the impression that the memory was happening here and now . They can be very short, elusive and fragmented’.27For each CB-IM, participants will be asked to write in a diary the time it occurred, its content and to report, from 0 = w nowness and (3) re-experiencing, avoidance, negative cognitions and mood, hyperarousal). Original instructions of the CBTS will be adapted for participants who had multiple childbirth(s) to complete the questionnaire regarding the index traumatic childbirth. Good psychometric properties were found for the French version of the CBTS.45This questionnaire assesses CB-PTSS during the past week, and diagnostic criteria of PTSD according to the DSM-5.This subscale assesses anxiety symptoms and consists of seven items using 4-point Likert scales.47This 10-item questionnaire measures postnatal depression symptoms during the past week with 4-point Likert scales.49Participants will answer the Feedback questionnaire III assessing intervention acceptability, usefulness, and expectations regarding its effects. These questions, based on previous work,Participants’ HRV will be recorded using a Firstbeat Bodyguard 2  ECG device during the whole intervention procedures. Count aloud tasks A and B will measure HRV baseline and recovery, in order to compare participants HRV during Narrative tasks A and B, and Tetris gameplay.not stressed and/or anxious at all to 10 = extremely stressed and/or anxious.Visual analogue scales will be presented to participants at 11 timepoints during the intervention to assess their subjective distress level from 0 = A sociodemographic questionnaire will assess participants’ age, nationality, marital status, education level and trauma history.Obstetric and medical data will be extracted from participants’ medical record, such as childbirth mode, gestational age and Apgar scores. Birth-related information will also be collected at baseline, through a self-report questionnaire .This study includes other outcomes, aiming to specify the effects of the intervention, presented in A sample size calculation was conducted for our primary and secondary outcomes based on the results obtained from the pilot study,Participants’ entries in electronic diaries and questionnaires will be directly stored in the REDCap software. The research staff will enter manually the rest of the data collected during the screening, the intervention and paper diaries in REDCap. All data will be hosted on a high security server of the Lausanne University Hospital performing automatic continuous backups. All primary outcome data and a random 5% of the rest of the data will be double-checked. Independent and blind coders will inspect diary entries to detect any non-compliance with instructions and count the number of CB-IM. A blind statistician will perform the analyses. To ensure blinding of the coders and the statisticians, they will receive a dataset not indicating which allocation group corresponds to the IIG or WCG. Only the study team will have access to the full coded trial dataset.For primary outcomes, group differences regarding change of the number of CB-IM between baseline and the 2 weeks postimmediate intervention will be analysed using repeated-measures analyses . For the secondary outcomes, repeated-measures analyses will be performed to analyse group differences concerning CB-PTSS between baseline and at 2 weeks postimmediate intervention. Logistic regression analyses will also compare the proportion of participants meeting the diagnostic criteria for CB-PTSD across time and groups. Furthermore, repeated measures analyses will serve to analyse change in the number of CB-IM and CB-PTSS between baseline and follow-up in the IIG. Group differences regarding the proportion of participants meeting the diagnostic criteria for CB-PTSD will be analysed with logistic regression analyses. The mean score of the intervention acceptability will be calculated across both groups. Additional post-hoc analyses conducted will be described as such in publications. Missing data will be dealt with using multiple imputation procedures, if appropriate.Any serious adverse event occurring during data collection will be documented and reported to the competent ethics committee.Participants will evaluate the intervention acceptability and will be able to give feedback on the intervention. Each participant will receive a personalised debriefing session. Study results will be sent to participants and shared with the public via social media and public events.Ethical approval of this protocol (dated 1 December 2022) was granted by the Human Research Ethics Committee of the Canton de Vaud (study number 2022-00652). The study sponsor is the Bureau du Promoteur de Recherche of the Lausanne University Hospital. Participants will provide an informed consent prior to study participation see . The stu10.1136/bmjopen-2023-073874.supp1Supplementary data


When dealing with a newly emerging disease such as COVID-19, the impact of patient- and disease-specific factors  on the immediate course of the disease is largely unknown. An accurate prediction of the most likely individual disease progression can improve the planning of limited resources and finding the optimal treatment for patients. In the case of COVID-19, the need for intensive care unit (ICU) admission of pneumonia patients can often only be determined on short notice by acute indicators such as vital signs , whereas statistical analysis and decision support systems that integrate all of the available data could enable an earlier prognosis. To this end, we propose a holistic, multimodal graph-based approach combining imaging and non-imaging information. Specifically, we introduce a multimodal similarity metric to build a population graph that shows a clustering of patients. For each patient in the graph, we extract radiomic features from a segmentation network that also serves as a latent image feature encoder. Together with clinical patient data like vital signs, demographics, and lab results, these modalities are combined into a multimodal representation of each patient. This feature extraction is trained end-to-end with an image-based Graph Attention Network to process the population graph and predict the COVID-19 patient outcomes: admission to ICU, need for ventilation, and mortality. To combine multiple modalities, radiomic features are extracted from chest CTs using a segmentation neural network. Results on a dataset collected in Klinikum rechts der Isar in Munich, Germany and the publicly available iCTCF dataset show that our approach outperforms single modality and non-graph baselines. Moreover, our clustering and graph attention increases understanding of the patient relationships within the population graph and provides insight into the network’s decision-making process. The exponential surge in cases overwhelmed intensive care units (ICUs), presenting scenes that had never been witnessed in the age of modern medicine3. During such a state of emergency, optimizing the allocation of hospital resources, e.g., ICU beds, mechanical ventilators, or personnel, becomes crucial. An essential aspect of effective patient management is correctly assessing treatment necessity and potential outcomes. When there is only a limited understanding of a previously unknown disease paired with highly multimodal data, as in the case of a novel pandemic, performing such an assessment and prediction of patient outcomes is very challenging. The resulting sudden overload of care facilities, in combination with the high complexity of the obtained data structure, motivates the need for assistance systems for fast outcome prediction and triaging based on available patient information. At the start of the COVID-19 pandemic, upon a patient’s hospital admittance, a multitude of parameters—such as sex, age, body weight, symptoms, co-morbidities, blood cell counts, inflammatory parameters, biochemical values, cytokine profiles, among others—were obtained and documented4. These parameters—“tabular data” in the following—and radiological images, including radiographs or X-ray computed tomography (CT) images, were available within the first hours after a new patient arrived at the hospital. This deems the two data sources ideal for early triaging and outcome prediction. Traditional disease outcome prognosis performed by clinicians is based, however, also on anamnestic information and clinical experiences. The general logic of a physician’s decision-making process partly relies on the information embedded in similar patients where the outcome and the connection of this information to currently treated patients are known5. Such population relationships are particularly useful when little to no disease-specific epidemiological information, as well as deep medical expertise, is available, as might be the case in potential upcoming health crises. Following this method of reasoning, we propose a decision support system that performs multimodal data analysis to create a population graph that clusters patients which is then used in a graph neural network with an attention mechanism to refine the patient outcome prediction by taking into account similar patients. The used similarity metric, attention mechanism, and generated pathology segmentations provide added insight into the decision-making process. This becomes possible as the weighting of clinical features and the most influential patients used in the prediction process can be directly observed. Our contributions are as follows:We introduce U-GAT, an end-to-end, graph-based method for leveraging medical images, extracted radiomics, and clinical data for predicting patient outcomes. In this work, we use multimodal data to predict COVID-19 patient outcomes, namely ICU admission, need for ventilation, and mortality. This method is generalizable and can easily be adapted to different types of anatomies, modalities, and clinical tasks.6 is used to segment the healthy and pathological regions of the lung in chest CTs. From these segmentations, we extract scalar values, in the following called “radiomics”, e.g., the percentage of healthy or pathological lung tissue volume, and subsequently perform a joint feature fusion of image, radiomic, and clinical features. This combined feature vector is refined in our Graph Attention Network (GAT)7 by leveraging similar patients to perform the final outcome prediction.Our model uses a multitasking approach, where segmentation and classification are learned simultaneously. A U-NetWe present an interpretable, multimodal patient similarity metric for graph construction and effective batch selection.We introduce a novel equidistant image sampling method allowing for end-to-end training of volumetric image feature extraction in a graph convolutional setting with multiple patients per batch graph. At test time, we make use of all available slices.We thoroughly evaluate our novel approach on a newly acquired dataset collected in Klinikum rechts der Isar in Munich during the first COVID-19 wave of 2020 as well as an external and publicly available dataset and showcase our model’s ability to predict patient-specific disease outcomes. The dataset from Klinikum rechts der Isar contains expert annotations of a diverse range of COVID-19 pathologies and is available for research purposes upon request.While we validate our method on COVID-19, it is disease-agnostic and the insights about modeling multimodal data without prior experience with patient trajectories can be easily adapted to new contexts of novel disease outbreaks.Reflecting on the coronavirus disease 2019 (COVID-19) pandemic8. A Feature-wise Linear Modulation (FiLM) layer affinely transformed the output of a Convolutional Neural Network (CNN) with a learned scaling and shifting factor using the text of the input question. Dynamic Affine Feature Map Transform (DAFT)9 extended FiLM to combine the features of 3D brain T1-weighted MRI scans and non-imaging biomarkers for Alzheimer’s prediction. DAFT affinely transformed the imaging features extracted by a 3D Fully CNN by a learned scaling and shifting factor using nine non-imaging features, such as age, sex, and genetic factors. A multi-headed cross-attention block has been recently proposed to fuse imaging and tabular data for skin lesion classification using a transformer architecture10 showing marginal improvement over joint fusion.Within the field of multi-modal learning, within recent years, different works have been published. One interesting approach to interweave features from multiple modalities was introduced by Perez et al. for visual reasoning tasks11 introduced ContIG, a self-supervised pre-training approach trained on 500k individuals from the UK Biobank12 combining retinal fundus images with genetic information tested on different classification and segmentation downstream tasks. A contrastive loss based on cosine similarity was utilized to decrease the distance of the embeddings of the multimodal features of one patient. Moreover, Duanmu et al.13 combined breast MRI scans and clinical biomarkers to predict chemotherapy response. A network trained on the non-imaging data learned scalar weights that were multiplied with the intermediate results from the imaging network to generate feature maps containing interactive information between imaging and tabular data.Taleb et al.14, there is a growing interest in developing similar machine learning systems. Huang et al.14 outlined three methods for integrating features in deep learning models for radiology: merging extracted image features with non-imaging features (early fusion), combining features with a joint end-to-end (image) feature extraction (joint fusion), and consolidating predictions made by independent models (late fusion). Our method employs joint fusion. In contrast to early and late fusion, joint fusion processes the different modalities separately but integrates them during intermediate stages, allowing for inter-modal interactions and joint model training. Backpropagating the loss function to the feature extraction allows the model to optimize the feature extraction based on the final output or prediction error, ensuring a more synergistic learning process. In the following, we review the fusion methods in the context of COVID-19.Inspired by the holistic decision-making approach taken by experienced physicians and medical boards, which involves integrating knowledge from diverse fields of expertise20. Chassagnon et al.21 demonstrated the importance of combining a wide range of non-imaging and extracted imaging features for the outcome prognosis of COVID-19 patients in an ensemble of machine-learning models. Shiri et al.22 achieved the best results in COVID-19 survival prediction by combining lesion-specific radiomics and clinical data. Gong et al.23 improved the results for predicting severe COVID-19 outcomes by adding blood values to other clinical features and extracted radiomics.For the COVID-19 detection and the prediction of patient outcome, most of the proposed methods integrating both imaging and non-imaging data apply early fusion of features24 reported an improvement in both COVID-19 severity and mortality outcome prediction compared to the stand-alone lung CT CNN and non-imaging Multilayer Perceptron (MLP) models. Tariq et al.25 explored different fusion methods for predicting the need for hospitalization of COVID-19 patients and found the early fusion of different electronic medical record features to work best for this task.Applying late fusion with penalized logistic regression, Ning et al.To the best of our knowledge, we are the first to propose a joint fusion method combining imaging and non-imaging data to predict ICU admission, ventilation, and mortality, or severity, depending on the dataset used.26 pioneered using GCNs on population graphs to improve Alzheimer’s and Autism Spectrum Disorder prediction. They also demonstrated that varying the patient information included in the graph setup significantly affects network performance5. Later works sought to diminish performance dependencies on graph generation, with Anirudh et al.27 suggesting a bootstrapping strategy and ensemble learning for GCNs. Cosmo et al.28 introduced a self-learning method for graph construction, integrating both imaging and non-imaging data for optimized GCN learning behavior. Further, GCNs have also been employed in medical image segmentation32 and Graph Attention Networks (GATs)7 have been utilized for patient diagnosis34.Previous studies have showcased the potential of Graph Convolutional Networks (GCNs) in medical applications, particularly in optimizing the processing of medical image information. Parisot et al.35 proposed a methodology that used GCNs on image data directly. They showed that end-to-end processing of imaging and clinical data within a GCN can improve performance due to optimized feature learning. At the same time, the proposed approach allowed for more effective usage of inter-class connections within the graph. We will expand upon this concept within our developed methodology and explain the implications in detail in section “The aforementioned works leveraged already extracted image features. However, Burwinkel et al.36 and Yu et al.37 built graphs based on the similarity of extracted CT image features and classified the nodes for the presence of infiltrates. In addition to image features, Song et al.38 and Liang et al.39 used the acquisition site along with other features to improve COVID-19 detection. Instead of modeling a patient population, Saha et al.40 converted edges detected in chest CT and X-ray images to graphs and leveraged these for detecting COVID-19. Huang et al.41 used GCNs to refine the segmentation of COVID-19 infections. Finally, Di et al.42 learned an uncertainty-vertex hypergraph to distinguish between community-acquired pneumonia and COVID-19. To the best of our knowledge, we propose the first graph-based end-to-end patient outcome prediction method by leveraging a population graph combining chest CTs and tabular patient data.In the context of COVID-19 diagnosis, GCNs have mainly been adapted for disease detection. Wang et al.44 on the radiological assessment of COVID-19 patients have shown a high correlation between disease burden and patient outcome, e.g., the probability of ICU admission. Several deep learning methods have been proposed to exploit this correlation with multitasking approaches47. The majority of the proposed multitask methods focus on the joint detection of COVID-19 infection and the binary segmentation of related pathologies in lung CT images52. Concerning COVID-19 patient outcome prediction, another set of works applied to multitask learning on the joint estimation of the severity of COVID-19 and various classification and segmentation tasks54. Similar to our approach, Nappi et al.55 used bottleneck features of a pretrained U-Net to predict COVID-19 progression and mortality. However, they did not optimize end-to-end, incorporate clinical patient data, or utilize a graph-based approach for the classification.Recent worksR that represent the relative burden of the lung for each pathology class. To effectively incorporate the different modalities, we introduce a new framework that combines the segmentation capabilities of U-Net with the analytic strengths of GCNs. This network uses a population graph constructed with the similarity of clinical patient data R to refine the image features of each patient. The proposed method operates end-to-end to perform an ideal combination of image feature representation learning, U-Net image segmentation, and graph data processing. The graph is pre-computed before training, and at test time, patients are dynamically connected to the graph of patients in the training set to ensure no data leaking during training and allow for usage flexibility in a clinical setting.Our proposed method provides an effective way to process multimodal patient information such as CT images mentclass2pt{minim35 showed that the image feature extraction process can potentially benefit from an underlying graph structure through an end-to-end feature extraction with a graph neural network since relevant graph information can backpropagate into the learned extraction process. We leverage this concept for the processing of the provided CT image information. Every CT image To allow for inference on unseen data samples, we employ spatial graph convolutions. Compared to spectral methods, this approach allows an extension to unseen samples, not requiring retraining for every new patient. As explained in section “S slices per volume along the axial view during training. If the main axis length is Z, each volume is divided into S slices, omitting Utilizing GCNs for end-to-end feature extraction from high-resolution 3D images presents a major challenge due to high memory demands, which restricts the number of patient instances per batch. However, GCNs necessitate diversity in a single batch for effective feature aggregation. To accommodate larger batches, we suggest equidistant subsampling of G with vertices V and connecting edges E. Every vertex R into one tabular feature and calculate the distance k nearest neighbors. As an alternative to feature selection, we propose to weight each feature based on a statistical analysis of the training data. Statistically important features should therefore have a bigger influence on the distance and similarity calculation. Possible weightings include correlation coefficients, e.g. the Pearson correlation for continuous features, or estimated mutual information56 between the input features and the target labels like We define a binary, directed graph entclass1pt{minima6 with small adaptions for our experiments. The S equidistant slices forming an input image c and the spatial dimensions c per slice. The resulting S slice-wise image representations are then transformed into a single patient-wise representation. To achieve this, the slice features are aggregated by taking the element-wise maximum along the stacking dimension resulting in a single vector with size c. This vector is then passed through a final fully connected layer followed by a leaky ReLU activation to obtain the latent image representation 54 using the final feature map of the U-Net instead of the bottleneck, we evaluated this approach, but initial results showed a substantial drop in performance which is why we did not investigate this concept any further.The proposed method is built on a joint image feature extraction and segmentation backbone. For this, any encoder-decoder-based architecture with a compressed bottleneck representation and segmentation output can be used. As described in more detail in section “4, the clinical data is complemented with radiomics features R that are automatically extracted from the segmentation output Inspired by Burian et al.Our methodology harnesses the multimodal data in a two-fold manner. On the one hand, radiomics extracted from the segmentation output and clinical patient parameters are employed to form the patient population graph. On the other hand, we synergistically fuse latent image features, extracted radiomics, and clinical data into the node features of this graph. This integrated representation encapsulates all salient attributes of a patient, providing a comprehensive patient characterization for subsequent processing. The three input sources provided by the image data 7. They combine effective processing of the provided neighborhood with the possibility for direct inference on new unseen data samples while maintaining filter localization and low computational complexity. The attention-based graph processing allows us to incorporate the clinical patient data N(i) for every The graph processing of our proposed method is based on graph attention layers (GAT)4. To assess the patient outcome, different parameters were collected: admission to the ICU, the necessity of mechanical ventilation, and the patient’s survival. These outcomes presented themselves immediately or sometime after general admission to the hospital. The complete dataset is available on request for research purposes in the frame of the BFS project AZ-1429-20C. For each CT volume, the total lung, healthy lung tissue, ground-glass opacifications (GGO), consolidations, and pleural effusions area were annotated by expert radiologists (4-8 years of experience). We combined pleural effusion and consolidation into a single class named “Other pathologies” since distinguishing between the two classes is a highly challenging task, even for senior radiologists58 as both have almost the same Hounsfield unit range. Moreover, pleural effusion is only present in the most severe cases in only 1.2% of all available patients. See radiomics statistics for this dataset in the The KRI dataset (“in-house” dataset) consists of 132 COVID-19 patients, expanding on the dataset with 65 patients described in24 . It comprises 1,521 patients and includes high-resolution CT images, clinical data, and patient outcomes. The main difference to the KRI annotations is the lack of image annotations of different pathologies in the lung. Since our work focuses on triaging patients infected with COVID-19, we exclude the control group and only predict the outcome severity of PCR-positive COVID-19 patients. This results in 620 patients with mild (Type I) and 274 patients with severe outcomes (Type II)24 leading to a total of 894 patients. Since the iCTCF dataset does not contain any annotations of the CT images, we employ a U-Net, pretrained on a diverse dataset61 of lung CT-slices by Hofmanninger62, to generate lung masks and a nnU-Net by Isensee et al.63, pretrained on the COVID-19 Lung CT Lesion Segmentation Challenge64, to infer the pathology annotation. The radiomic COVID-19 burden was extracted using this annotation, resembling the percentage of the lung affected by COVID-19 pathologies.To substantiate the versatility of our method, we have extended our evaluation to a larger and publicly available dataset: the iCTCF dataset65 stratified by the ICU labels. For this, the dataset is split into five equally sized folds, each containing a similar amount of ICU patients. In nested cross-validation, there are outer and inner evaluation loops for testing and validation. In each of the five outer loops, one fold is selected as a test set, and the remaining four folds are used for training and validation. In the four inner loops, three folds are selected for training and one for validation. This is repeated until every combination has been used for testing and validation, resulting in a total of 20 repetitions.We first evaluate the proposed method on the KRI dataset using a nested 5-fold cross-validation4, the static lung CT images taken at admission were used in combination with the following clinical features and blood test results: age, sex, body temperature, percutaneous oxygen saturation, leukocytes, lymphocytes, C-reactive protein (CRP), creatine, D-Dimer, lactate dehydrogenase (LDH), creatine kinase, troponin T, interleukin 6 (IL-6), thrombocytes. The outcomes included: the need for mechanical ventilation, admission to the ICU, and patient survival . All three tasks are binary classification tasks. We focus on evaluating the main task of ICU prediction and extend some experiments on ventilation and mortality outcome tasks to explore multitasking and the translation to other tasks. The experiments were conducted with ten equidistant samples  loss for classification. Further training details can be found in the 7.We conducted all experiments in PyTorch 1.7.056 with 3 neighbors averaging the results of 30 repetitions. We compared weighting the KNN with mutual information against weighting with Pearson correlation. To understand the impact of weighting features, we also compared these weighted methods against an unweighted KNN. For the unweighted setup, we evaluated different subsets of manually selected features as can be seen in Table k used for graph construction was set in a hyperparameter search on the validation set.We employed the KNN graph construction method introduced in section “69. It is important to note that radiomics were not used in the ResNet18-GAT architecture because ResNet18 does not produce segmentations. To evaluate the contribution of GAT, we compare it with the following classification method alternatives:70.Weighted K-nearest neighbors (KNN): The default scikit-learn weighted k-nearest neighbor classifier using the inverse Euclidean distance of all features as the similarity metric for neighbor selection and for weighting of neighbor labelsMultilayer Perceptron (MLP): This classifier is a simple neural network with a hidden layer size of 64 followed by a leaky ReLU activation and a 10% dropout.57.GraphSAGE: replacing the GAT operator with GraphSAGEIn addition to ablative testing, we compare unimodal vs. multimodal approaches by evaluating the performance of using an MLP classifier using only clinical data or only image features extracted by a ResNet18. An overview of the type of data used in each method is given in Table To investigate the effect of the different components of our method, we show ablative results on the test set. We mainly evaluate two components: the image and radiomics feature extraction of the U-Net and the GAT classification. The end-to-end U-GAT feature extraction is compared with features extracted from a simple frozen U-Net trained on the same annotations but without any multi-tasking, and the end-to-end image features from a ResNet18 as proposed by He et al.71 and provide the additional benefit of interpretability. As discussed in section “4 and Chao et al.15 have successfully deployed Random Forests to use tabular radiomics and clinical data for ICU prediction. In this experiment, we focus on the task of ICU prediction and explore if an ensemble of our proposed model can improve its performance due to increased robustness against overfitting and how it compares to the well-established Random Forest classifier. To form an ensemble we average the predicted probabilities of the 4 models trained on the inner loops of the nested cross-validation and evaluate them on the 5 test sets of the outer loop of the nested cross-validation.Random Forest is an ensemble method that is an effective classifier for small datasets since they are less prone to overfitting due to the Law of Large Numbers72: 70.As our proposed method follows a multitask approach including the CT segmentation and each of the tasks of ICU, ventilation and mortality prediction individually, the evaluation criteria can be divided into segmentation and classification metrics. To measure the overlap between segmented regions and ground truth, we use the Dice score (DS). The main metrics for evaluating the binary classification performance are average precision (AP) and the area under the receiver operating characteristic curve (AUC), as they are independent of selected classification thresholds. Given that all tasks have a severe class imbalance, the F1 score (F1) has been chosen as the main threshold-dependent metric. In the ensemble experiments, the balanced accuracy score (bACC), sensitivity, and specificity are additionally reported. For all threshold-dependent metrics, the optimal threshold is set using the validation results and maximizing the Youden’s J statisticIn the first phase of experiments on our KRI dataset, we optimized the population graph construction method. This involved evaluating various feature selections and distance weights to improve the KNN-based graph construction. We found that connecting each node with its seven nearest neighbors provided optimal results, based on a hyperparameter search using a simple, unweighted KNN classifier. Two measures - mutual information and Pearson correlation - were used to weight features in the distance calculation of the similarity metric used for KNN neighbor selection. Table hand see . The perbels see . A key bentclass1pt{minimaIn the next set of experiments shown in Table The results of joint end-to-end training of the segmentation and classification task seem to improve the AP slightly for all tasks on both datasets. While the average Dice score is lower in all multitask setups than in the segmentation single-task setup see , this ma outcome . This inWe conducted additional experiments on the synergistic effects of multitasking segmentation with classification and the concurrent prediction of different patient outcomes since all of these tasks are interdependent. Results detailed in the As discussed in section “N is the number of nodes in the batch and all rows in A add up to 1. We can multiply the attention matrices of both layers to receive a matrix that shows how the representation of a node is based on its two-hop neighborhood, i.e., all nodes that are at most two edges away. These attention scores are visualized in Fig. In addition to its performance boost over GraphSAGE, using GAT offers another important advantage. The attention mechanism of our model learns to identify the neighbors in the graph that are the most relevant for the prediction task, providing insight into the decision process of the model. The analysis of attention scores could suggest patients that the model deems relevant for the individual outcome prediction. These connections within the patient population graph can help uncover new information about a disease that is still poorly understood and provide valuable insights to physicians. Combined with the segmentation results, our attention mechanism allows the clinicians to thoroughly evaluate our model output and decision-making process, giving them potentially higher confidence in the prediction. For each of the two GAT layers, the model assigns attention scores to the neighbors of each node in the graph. These scores define how much the node representation after the layer will be based on the representation of its different one-hop neighbors. These attention scores can be thought of as a weighted directed adjacency matrix In a future iteration of our current model, the segmentation of infrequent lung pathologies, such as pleural effusion could be improved along with the prediction of imbalanced outcomes, notably mortality. Our approach to enhance the model involved constructing the population graph based on the mutual information of each feature. This has effectively improved the graph structure, and importantly, the features identified through this method are consistent with established radiological findings. It should be noted, however, that the mutual information displays a pronounced standard deviation and is notably lower for the mortality prediction task, indicating the inherent complexity of this specific prediction task and the potential sparsity of highly informative features given the available parameters in the dataset. In subsequent studies, these areas can be addressed by incorporating more annotated data and expanding the patient cohort, particularly the clinical data.In this work, we developed and evaluated a method to effectively leverage multimodal information for the outcome prediction of COVID-19 patients. Here, the said information in the form of CT lung scans, clinical data, and radiomics was incorporated into a graph structure and processed within a GAT to stabilize and support the prediction based on data similarity. With U-GAT, we propose an end-to-end methodology that segments patient pathologies in medical images and uses a combination of imaging and non-imaging data to predict clinical outcomes. We explicitly incorporate automatically extracted lung radiomics in our architecture and demonstrate increased performance. We show that the auxiliary segmentation of COVID-19 pathologies indeed improves outcome prediction. To create the patient population graph, we propose a novel graph construction based on feature weighting utilizing mutual information, effectively clustering relevant patients. Our attention analysis imparts an additional layer of transparency, potentially increasing clinicians’ confidence in our predictive approach. This added clarity can assist in identifying comparable patients from previous cases, thus informing and guiding the treatment trajectory for the current patient under consideration. This study underscores the potential of graph-based, data-driven strategies in improving patient care and decision-making in challenging clinical settings using multiple modalities.Supplementary Information.


To assess if incomplete meniscal healing during second-look arthroscopy at six to eight weeks after all-inside suture hook meniscus repair results in longer-term failure of repair in patients with restored knee stability.From 2008 to 2013, 41 patients with post-traumatic, longitudinal, vertical, complete meniscal tears with concomitant ACL injury were treated via a two-stage surgical procedure and prospectively evaluated. In the first stage, all-inside meniscus repair was performed using suture hook passers and non-absorbable sutures. In total, there were 26 medial and 16 lateral meniscus tears. A second-stage ACL reconstruction, performed six to eight weeks later, served as an early second-look arthroscopic evaluation of meniscal healing. Clinical follow-up was performed at a minimum of 24 months.p = 0.012). One saphenous neuropathy occurred (2.4%).Second-look arthroscopy revealed 31 cases (75.6%) of complete and ten cases (24.4%) of incomplete meniscal healing. Two patients were lost prior to follow-up, and three were excluded due to recurrent instability. Therefore, 36 patients were assessed at the final follow-up. All patients with complete meniscal healing during second-look arthroscopy achieved clinical success at follow-up. Six out of nine (66.7%) of patients with incomplete meniscal healing during second-look arthroscopy achieved clinical success at follow-up (Incomplete meniscal healing during early second-look arthroscopy after all-inside meniscal repair using suture hook passers and non-absorbable sutures did not necessarily result in longer-term failure in patients with restored knee stability. The described method of meniscal repair was associated with a low rate of symptomatic re-tears and complications. Post-traumatic meniscal tears are commonly seen in young patients as a result of acute, sports-related trauma. These tears are typically longitudinal and vertically oriented along the peripheral third of the meniscal body and are commonly associated with knee instability due to concomitant anterior cruciate ligament (ACL) deficiency , 2. In pSuccessful meniscal repair depends on a successful healing which is based on two fundamental principles: solid mechanical fixation and the biological healing process . The menThe aim of this paper was to assess if incomplete meniscal healing during second-look arthroscopy at six to eight weeks after all-inside suture hook meniscal repair results in longer-term failure of the repair in patients with restored knee stability.Ethical approval was obtained from the district medical chamber . The study was designed in compliance with the Helsinki Declaration.This was a retrospective cohort study of consecutive patients presenting to a single centre with post-traumatic, complete, vertical, longitudinal tears of the outer third of the meniscus and a concomitant anterior cruciate ligament (ACL) injury. During the period from 2008 to 2013, 41 consecutive patients  who underwent a two-stage surgical procedure were included in the study, and all data were prospectively collected into a database. Exclusion criteria were (1) meniscal body degeneration, (2) multi-ligament instability and (3) chondral injury > stage II according to the International Cartilage Research Society (ICRS) classification .As summarised in the study timeline diagram Fig. , the firThe first stage consisted of an all-inside meniscal repair using suture hook passers and non-absorbable sutures, without medial collateral ligament (MCL) release, as elaborated in the “Methods and materials” section, under the heading Surgical Technique of Meniscal Repair Fig. .Fig. 2MeAfter six to eight weeks, the second stage surgery was performed, which comprised an ACL reconstruction (ACLR) using quadriceps tendon-bone (QTB) autografts and served as a second-look arthroscopic evaluation of meniscal healing Fig. . The heaClinical follow-up was performed at a minimum of 24 months Fig. . PatientThe surgery was performed under either general or regional anaesthesia. A thigh tourniquet was not used to allow intra-operative observation of bleeding from the meniscus. Routine diagnostic arthroscopy was performed through standard anterolateral and anteromedial portals using a 30° arthroscope. Accessory medial parapatellar and additional posterolateral or posteromedial portals were made to facilitate visualisation and manoeuvrability within the posterior compartments. The diagnosis of medial meniscus (MM) or lateral meniscus (LM) injury was made by direct visualisation and probing with an arthroscopic hook probe.After confirmation of a complete vertical longitudinal tear of the outer third of the meniscus, it was repaired via a previously described all-inside technique using suture hook passers  and three to eight non-absorbable sutures  depending on the length of the tear , 12. FirFor post-operative management, patients were instructed to perform 5 min of passive hyperextension and 5 min of passive flexion every hour for the first two weeks after the meniscus repair surgery. Range of motion (ROM) up to 90° of knee flexion was permitted starting on postoperative day one. During the first six weeks, the patients were limited to walking only while using crutches, with a decrease of walking volume. For the first two weeks, only touch-weight bearing was permitted, which was thereafter progressively increased according to the patients’ tolerance of pain for weeks three to six. No twisting or pivoting movements were allowed. Guided physiotherapy was performed from the second week postoperatively. Symmetric hyper-extension and 120° of flexion had to be achieved before progressing onto the second operative stage.The data were analysed statistically using Statistica 13.3 software. A two-sided Fisher exact test was used to compare the rate of clinical failure between patients with complete and incomplete healing during second-look arthroscopy. Calculation of relative risk (RR) or odds ratio (OR) was not possible due to 0% clinical failure rate in one of the groups.The diagnostic assessment of the first-stage procedure confirmed 15 cases of “bucket-handle” tears, ten of the medial meniscus (MM) and five of the lateral meniscus (LM); 26 cases of posterior horn tears, 16 of the MM and ten of the LM and one case of an anterior horn tear of the LM . In total, there were 26 MM tears and 16 LM tears across the 41 patients. The results of diagnostic arthroscopy are summarised in Table As summarised in Fig. p = 0.012). The relationship between meniscal healing assessed during the second-look arthroscopy and clinical success at follow-up is summarised in Table All 27 patients within the complete healing group achieved clinical success at the final assessment. In the incomplete healing group, six out of nine patients achieved clinical success at follow-up. The rate of clinical failure at follow-up was significantly higher in patients with incomplete healing during second-look arthroscopy than in patients with complete healing  that exhibited only incomplete meniscal healing during second-look arthroscopy still achieved good clinical results . Therefore, it can be concluded that the early arthroscopic finding of incomplete healing does not necessarily result in longer-term failure of repair and does not allow for an easy decision of whether to perform any additional surgical interventions. These results show that while second-look arthroscopy is the gold-standard method for accurate assessment of meniscal healing, eight weeks may be too early to correlate the existence of incomplete healing of the meniscus at the repair site with long-term failure.The results of this study also demonstrate that in comparison to other methods of meniscal repair, the all-inside suture technique with the use of suture hooks yields successful clinical and functional outcomes. Several other studies have reported comparable clinical success using similar surgical techniques with the use of suture hooks. Ahn et al.  assessedWhile previous studies have reported on the success of meniscal repair in the setting of ACLR –20, thisWe theorised that the comparably successful meniscal healing rate despite two stages and the potential risk associated with instability prior to the second stage surgery could be connected with two possible reasons.First, during second-stage ACLR, the healing response could be once again promoted due to bone drilling, resulting in the release of growth factors from the bone marrow and haematoma formation , 21, 22.The limitations of this study were the varying time from injury to surgery and relatively small number of patients included in this study. As to the varying time from injury to surgery, while this limits the homogeneity of the group, it mimics real-life in-patient scenarios. Therefore, in another aspect, it may very well serve to strengthen the validity of the clinical results presented in this study. As to the second limitation, although the present study was limited by its number of subjects, several studies that had performed a similar surgical technique are presented, with comparable successful clinical outcomes. On the other hand, to the extent of the authors’ knowledge, this was also the first study to assess all-inside suture hook meniscal repairs using non-absorbable sutures.Incomplete meniscal healing during early second-look arthroscopy after all-inside meniscal repair using suture hook passers and non-absorbable sutures did not necessarily result in longer-term failure in patients with restored knee stability. The described method of meniscal repair was associated with a low rate of symptomatic re-tears and complications.


The anatomical infiltrated brain area and the boundaries of gliomas have a significant impact on clinical decision making and available treatment options. Identifying glioma-infiltrated brain areas and delineating the tumor manually is a laborious and time-intensive process. Previous deep learning-based studies have mainly been focused on automatic tumor segmentation or predicting genetic/histological features. However, few studies have specifically addressed the identification of infiltrated brain areas. To bridge this gap, we aim to develop a model that can simultaneously identify infiltrated brain areas and perform accurate segmentation of gliomas.N = 270), validation (N = 30), and independent test (N = 54) sets. We evaluated the predictive performance using the area under the receiver operating characteristic curve (AUC) and Dice scores.We have developed a transformer-based multi-task deep learning model that can perform two tasks simultaneously: identifying infiltrated brain areas segmentation of gliomas. The multi-task model leverages shaped location and boundary information to enhance the performance of both tasks. Our retrospective study involved 354 glioma patients (grades II-IV) with single or multiple brain area infiltrations, which were divided into training , a sensitivity of 87.67%, a specificity of 87.31%, and accuracy of 87.41%. Specifically, for grade II-IV glioma, the model achieved AUCs of 95.25% , 98.26% , and 93.83%  respectively for the identification of infiltrated brain areas. Moreover, our model achieved a mean Dice score of 87.60% for the whole tumor segmentation.Experimental results show that our multi-task model achieved superior performance and outperformed the state-of-the-art methods. The impressive performance demonstrates the potential of our work as an innovative solution for identifying tumor-infiltrated brain areas and suggests that it can be a practical tool for supporting clinical decision making. Gliomas are a type of malignant tumor that develops in the glial cells of the brain and spinal cord, and they are among the most lethal neurological malignancies , 2. The Magnetic resonance imaging (MRI) is a highly promising imaging technique due to its non-invasive nature. T2-fluid-attenuated inversion recovery (T2-FLAIR) abnormality is a reliable indicator of tumor progression, which has been found to correlate with impoved survival rate , 6. NumeHowever, most studies have focused on glioma segmentation for clinical applications, with relatively little attention given to automatic identification of the precise anatomical location of glioma within the brain. It should be noted that the anatomical location of a glioma plays a crucial role in determining treatment options, clinical courses, and prognosis , 15.A glioma tumor typically manifest in one or multiple brain areas , which To address the aforementioned issues, we propose a transformer-based multi-task deep learning model that achieves simultaneous glioma segmentation and identification of infiltrated brain areas in an end-to-end framework. The main contribution of our study is the ability to accomplish these tasks simultaneously by sharing local and global features extracted from a hybrid CNN and Transformer network. Specifically, our multi- task deep learning framework leverages the boundary information from segmentation to enhance the performance of identifying glioma-infiltrated brain areas.With the approval of institutional ethics committee, patients who received brain tumor resection and were diagnosed with glioma according to the 2016 WHO criteria  were retThe details of the demographic and clinical characteristics of patients, as well as the whole tumor cohort, are shown in Table To reduce the effects of variability in acquisition and sequence parameters, image pre-processing was applied before analysis, which included MRI offset correction, image registration, skull stripping, and gray-level normalization. The N4ITK algorithm  was adopThe architecture of our proposed transformer-based multi-task model for simultaneous infiltrated brain area identification and segmentation of gliomas is presented in Fig. The performance of the models was evaluated on an independent test set, which was not included in the model training. The remaining data were randomly divided into training and validation sets for parameter selection. After obtaining the optimal model parameters using the training and validation sets, the optimal model parameters were used to train the final model using the training data and validated on the independent test set.height) × 352 (width) × 20 (depth). Gray-level normalization [The models were trained using PyTorch v1.7, with a data batch size of 4 and input image dimensions of 416  and two clinical data variables (glioma grade and Karnofsky performance score) used in the comparison experiment were also pre-treated. Except sex was labeled with 0 and 1, the other three characteristics was standardized with z-score. Unlike image information, which is encoded using convolution, the binarized gender information is first transformed into one-dimensional features through Word Embedding. Prior to the fully connected layer, the three standardized clinical features , the gender features represented in the embedding space, and the image features are concatenated together and collectively fed into the fully connected layer for the final prediction.95) to evaluate the segmentation performance of the segmentation model. The DSC measures the degree of overlap between the ground truth and the predicted segmentation, with 100% indicate complete overlap.All well-trained models were evaluated on both a validation set and an independent test set. The performance of multi-label classification model was assessed using various metrics, including the area under the receiver operating characteristic curve (AUC), accuracy (ACC), sensitivity (SEN), and specificity (SPE). Micro-averaging is used to describe all indexes of multi-label classification. The optimal threshold for the AUC value was determined by maximizing the sum of the sensitivity and specificity values. The 95% confidence intervals (CIs) were obtained using bootstrapping to assess variability. Besides, we used the Dice score (DSC), accuracy, sensitivity, specificity, Intersection over Union (IOU), Average Symmetric Surface Distance (ASSD), and 95th Percentile Hausdorff Distance  with an average age of 47.61 ± 12.99 years (range from 11 ~ 82 years) were included in our study. Of these, 270 were allocated to the training set, 30 to the validation set, and 54 to the independent test set. As shown in Table Tables We also integrated clinical features into our model, specifically two demographic characteristics (age and sex) and two clinical data variables (glioma grade and Karnofsky performance score). The results presented in the table demonstrate an improvement of 2.73% in AUC, 6.21% in accuracy, 4.77% in sensitivity, and 6.8% in specificity compared to our original MR-only model for all grades on validation set.The performance of five different tumor segmentation methods, including U-net, nnUnet, COM-Net, MTTU- Net and our method was evaluated on the validation and independent test sets. The related results are presented Table To compare the accuracy of infiltrating tumor lobe classification by human and the model, we conducted a comparison experiment with a random sample of 50% in independent test set. Two experts dedicated to annotate glioma-infiltrated brain lobes based on enhanced T1 and T2-FLAIR MRI data . They consecutively and independently evaluated the MR data from independent test set. The results of our model have been binarized for fair comparison. Table To validate the extraction of specific global and local features, we compare the performance of the hybrid CNN and Transformer network with a pure CNN network using guided backpropagation  for visuThe pure CNN network shows attention to local tumor information, as evidenced by the high intensity in the tumor region (Fig. In this study, we have developed a method that can predict the glioma-infiltrated brain areas while simultaneously achieving the tumor segmentation. Our approach incorporated the CNN and transformer network to leverage both local and global features, thus enhancing the performance of classification and segmentation.Our method can automatically identify the tumor-infiltrated brain areas and provide a whole tumor segmentation, which can aid in determining the appropriate surgical approach and predict the prognosis , 15. WheThe precise characterization of glioma is critical for clinical treatment plans . PreviouIn contrast to molecular subtyping or grading of glioma, where each patient has a single label for a task, our infiltrated brain area identification task may have one or more labels per patient. Directly adapting the commonly used deep learning-based multi-class classification models, such as ResNet50 and VGG16, to the multi-label task may not be effective due to the irregular shape of gliomas. While adding multi-sequences of MRI images can help improve the performance of infiltrated brain area identification, the improvement is limited without the prior information about the glioma boundary. There are several factors that may explain our proposed method achieved higher performance of identifying glioma- infiltrated brain areas compared with other methods. First, unlike the aforementioned methods that perform classification and segmentation separately, our method uses a single encoder to extract the task-relevant features. Learning shared representations across different tasks can help prevent overfitting on each task and improve overall performance on all tasks. Second, our method co-optimizes the loss functions of brain area identification and tumor segmentation, improving the model's generalization ability and potentially solving label imbalance problems via shared label information. Third, transformer blocks were embedded into CNN-based network to address the intrinsic locality limitation of convolution operations.Despite the superior performance of the validation and independent test sets, our study has several limitations. The main drawback is its retrospective nature, which makes it vulnerable to selection and recall bias, thereby potentially impacting the reliability and generalizability of the study findings. Our future direction encompasses the implementation of prospective studies, enabling the longitudinal follow-up of patients from the onset and facilitating the collection of real-time data for precise and comprehensive analysis. This approach aims to enhance the validity and robustness of our model while augmenting its applicability in clinical settings. Besides, advanced MRI sequences, such as diffusion or perfusion weighted imaging MRI, were not included in this study. While these sequences are not the clinical imaging routine for structural scans. Our results in Tables In addition, we acknowledge the significance of detailed structural labeling, including deeper brain structures like the internal capsule. Future research will focus on enhancing the capabilities of our model to enable more accurate and detailed labeling of brain structure and function. We will incorporate advanced techniques and expand the scope of our model to encompass a broader range of anatomical and functional regions.In conclusion, we have developed a transformer-based multi-task deep learning model that can perform two critical tasks: identifying tumor-infiltrated areas of the brain and segmenting the tumor based on preoperative MR scans. Our model has demonstrated superior performance compared to state-of-the-art classification and segmentation models on both validation and independent test sets. It is believed that our model holds significant potential as a practical tool to assist clinicians in accurately identifying glioma-infiltrated areas and supporting treatment decisions.


As the first wave of SARS-CoV-2 infections spread across the globe there was a noticeable paucity of cases reported from countries in sub-Saharan Africa (SSA). Many theories were proposed to explain the epidemiology of COVID-19 in this region. One relates to population demographics; there is a clear positive correlation between age and risk of infection . The sixPlasmodium vivax . They tested 998 participants and only 8 were positive for both SARS-CoV-2 and P. falciparum (by PCR). All study participants were asymptomatic; thus, it was not possible to glean insight into the effect of malaria co-infection on the severity of SARS-CoV-2 or vice versa. Yet this question was addressed in the study from Mali reported by Woodford et al. They asked whether seasonal malaria affected SARS-CoV-2 seroconversion rates, clinical presentation, and antibody responses. Using a longitudinal study design, they evaluated the effects of recent or current malaria on SARS-CoV-2 infection. They found no evidence to support an effect of malaria infection on antibody responses to the virus or the severity of symptoms at presentation. These data strongly refute the hypothesis that malaria may protect against SARS-CoV-2.One intriguing idea raised by Konozy et al. is that SARS-CoV-2 may infect RBCs Woodford et al., research staff responded to the pandemic by expanding their research capacity to study the interactions between malaria and SARS-CoV-2. They also successfully introduced vital engineering controls, promoted staff training and education to enhance safety of staff and participants.The ability to perform this study in Mali at a time when community-based research was largely impossible is attributed to the research infrastructure already in place to conduct malaria clinical trials. The long-standing collaboration between US and Malian researchers meant that infrastructure established to support malaria trials could be leveraged for sub-studies on COVID-19. As discussed in the opinion article by DePina et al., the authors examined the impact of COVID-19 restrictions on care-seeking behaviour and the frequency of malaria testing using an interrupted time series analysis. Surprisingly, while there was a 28% decrease in the number of outpatient visits, there was a non-significant 8% increase in testing for malaria per 1,000 outpatient visits. As a result of sustained surveillance throughout the pandemic, the country now qualifies for elimination status.Early in the pandemic when lockdown measures were in full force, a pressing public health concern was that disruptions to essential malaria services would compromise control and surveillance efforts. In effect, 13.4 million additional cases and 63,000 deaths between 2019 and 2021 were attributed to the pandemic . This inThe articles in this Research Topic addressed the specific context in SSA where SARS-CoV-2 emerged in countries already battling malaria. The prospect of negative interactions between the two pathogens, where one could exacerbate the other, would be dire and further compromise malaria control efforts and management of COVID-19. Instead, we learned that the two pathogens seem to be independently pathogenic and seroconversion to SARS-CoV-2 is not impeded by past exposure to malaria. Most promising are the public health lessons on the remarkable resilience of research teams and communities to sustain malaria control efforts and clinical trials under these exceptionally challenging circumstances.All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.


BMC Ecol Evo 21, 118 (2021)Correction: 10.1186/s12862-021-01845-2Following the publication of the original article , a potenIn noting that many species of bdelloid rotifer display a shared, but variable-length deletion in the 18S rRNA molecule that could extend into a critical structural position , it was also cautioned in the original article that these deletions could obtain from a sequencing artefact. The latter has now been positively confirmed by one of the authors of the sequences in question  and the appropriate GenBank accessions will be amended to indicate this.


Optical approaches are useful for studying the electronic and spin structure of materials. Here, based on the tight-binding model and linear response theory, we investigate the magneto-optical Kerr and Faraday effects in two-dimensional second-order topological insulators (SOTI) with external magnetization. We find that orbital-dependent Zeeman term induces band crossings for SOTI phase, which are absent for trivial phase. In the weak-magnetization regime, these crossings give rise to giant jumps (peaks) of Kerr and Faraday angles (ellipticity) for SOTI phase. In the strong-magnetization regime, we find that two nearly flat bands are formed at the high-symmetry point of Brillouin zone of SOTI phase. These flat bands give rise to two successive giant jumps (peaks) of Kerr and Faraday angles (ellipticity). These phenomena provide new possibilities to characterize and detect the two-dimensional SOTI phase. Different from the conventional correspondence between d-dimensional bulk and  quantum Hall and quantum anomalous Hall effect24, spin transport25, etc. By contrast, two-dimensional (2D) SOTI has received relatively less attention due to the difficulties in the material growth and detection of higher-order topology28.In recent years, there has been a surge of interest in the topological properties of quantum materials. Amongst these, the concepts of topological invariants have been generalized to higher orders31 and experimentally observed34. Kerr and Faraday effects are not restricted to 3D bulk or film systems, but can be employed to 2D materials. For example, polar Kerr effect may provide fingerprints of spontaneously broken time-reversal symmetry in bilayer graphene35. Experimentally, giant Faraday rotations have been observed in monolayer graphene under modest magnetic fields37. Furthermore, magneto-optical Kerr effects have also been used to experimentally demonstrate the 2D ferromagnetic behaviors of monolayer CrI38 and Cr39. Since magneto-optical Kerr and Faraday effects can characterize the magnetism and spin behaviors of electrons41, it motivates us to study the topological properties of 2D SOTI by using these techniques.Optical measurements may provide efficient ways to detect the higher-order topology, as they are bulk sensitive and do not rely on the details of boundary states. When a light is incident into magnetic materials, its angular momentum is transferred to the reflected and transmitted light, respectively, giving rise to the rotations of polarization planes see Fig. . These c43 with some symmetry-breaking terms. The advantage of the model is that we can switch SOTI phase on and off by tuning parameters. This provides opportunities to compare the results of SOTI with trivial insulators. The light is normally incident into 2D SOTI and magnetic substrate from the vacuum, whose electromagnetic field  follows the standard Maxwell’s equations31. We relate the electromagnetic fields in the vacuum and substrate region by the modified boundary conditions incorporating the conductivities contributed by 2D SOTI. By solving these equations, the Kerr and Faraday angles are then directly obtained from the reflection and transmission coefficients of electric field. On the other hand, the finite-frequency longitudinal and Hall conductivities of 2D SOTI are derived by using the Kubo formula based on linear response theory44. Particularly, the Hall conductivity tensor is a consequence of out-of-plane magnetization in 2D SOTI.In this work, we study the magneto-optical Kerr and Faraday effects in 2D SOTI with out-of-plane magnetization. We consider the generic tight-binding model of 2D SOTI, constructed by the model of 2D topological insulators43, which can be decomposed into orbital-independent and orbital-dependent terms. By symmetry analysis, we find that only orbital-dependent Zeeman term contributes to the Kerr and Faraday effects in such systems. We also find that the magnetization induces band crossings in conduction and valence bands only for the SOTI phase. In the regime of weak magnetization, these crossings lead to giant jumps (peaks) of Kerr and Faraday angles (ellipticity). In the regime of strong magnetization, two nearly flat bands are formed at the high-symmetry X point of Brillouin zone of SOTI. These give rise to two successive giant jumps (peaks) of Kerr and Faraday angles (ellipticity) for the SOTI phase. By the quantitative analysis, we find that the model parameters and order of magnitude of rotation angles are all within experimental reach for realistic materials. Therefore these phenomena provide new features to characterize the SOTI phase, which may have practical applications in distinguishing SOTI from trivial insulators.For the treatment of magnetization, we consider the Zeeman effect in multi-orbital systems13, witha is the lattice spacing (set to be unity). Pauli matrices 42. 43. Other parameters such as M and B are discussed in the units of t in the following. We consider a generic tight-binding model of two-dimensional chiral second-order topological insulators 43, the Zeeman term readsg factors of electronic orbitals In this paper, we consider the external magnetization-induced rather than magnetic-field-induced Kerr and Faraday effects. Thus there are no Landau levels and the only consequence of magnetization is the Zeeman energy. For the HgTe quantum wellsmentclass2pt{minimThe symmetry properties of the Hamiltonian mentclass2pt{minim44The optical conductivity tensor can be given by using the Kubo formulamentclass2pt{minimWhen the orbital-independent Zeeman term mentclass2pt{minim314546When a light is propagating along mentclass2pt{minim48 when 50. The energy dispersions of Hamiltonian 52 or monolayer MoTe53. The photon energy ranges from 0.01 eV to 0.6 eV, corresponding to the terahertz and far infrared frequencies54. In the weak magnetization regime, the rotation angles are tens of mrad, which share the same order of magnitude with experimental results of Bi32. In the strong magnetization regime, the rotation angles become a few mrad, in the same order of magnitude with experimental results of strained HgTe and Bi34. Our studies can also be applied to other proposed 2D SOTI, such as graphdiyne26, Bi on EuO substrate27 and monolayer FeSe28.Numerical results are mainly based on the model parameters of HgTe quantum wells. x and y direction. The other is to induce (To realize SOTI from magnetic doped TI, the existence of mentclass2pt{minimmentclass2pt{minim31, we know that the results are independent of the substrate properties when the substrate thickness is much smaller than the light wavelength in the low-frequency limit. By increasing the thickness, the magnitude of Kerr and Faraday angles is suppressed. Particularly, when the resonance conditions are satisfied, that is, the substrate thickness contains an integer number of half wavelengths, Kerr and Faraday angles show Fabry-Perot-type oscillations and again become independent of the substrate properties.Note that in our setup see Fig. , semi-inSince ellipticity is a dispersive effect and sensitive to the distortions of dielectric tensor, it may not be suitable to detect 2D SOTI. Ellipticity may serve as a supplementary to Kerr and Faraday angles, which together can be used to characterize the 2D SOTI phase. Additionally, the comparison between ellipticity and rotation angles can provide information of distortions or inhomogeneities of the system.30 and Floquet topological insulators55 . By contrast, we are concentrated on the SOTI with proximity magnetization, rather than introducing the Landau levels55. Rashba spin-orbit interaction may further be introduced at the interface as a result of inversion symmetry breaking. This may modify the spin and pseudospin structures of electronic bands, inducing additional channels for optical interband transitions. The presence of impurity scattering affects the relaxation time 31. For surface states of topological insulators Bi57, hexagonal warping term is present. This term may modify the interband transitions, Fermi velocity and density of states, leading to a quasilinear shape of 59. Our discussion focuses on the zero-temperature limit, and the increasing temperature tends to suppress the magnitude of peaks and jumps of Kerr and Faraday rotations33. However, as long as the temperature is not high enough, the main features should still be observable.Magneto-optical Kerr and Faraday effects have also been studied in topological insulatorsX point for SOTI. These give rise to two successive giant peaks of To conclude, we have studied the magneto-optical Kerr and Faraday effects in two-dimensional second-order topological insulators. By symmetry analysis, we find that to observe the Kerr and Faraday effects in such systems, Zeeman term must be orbital dependent, rather than orbital independent. The magnetization induces new crossings in conduction and valence bands only in the SOTI phase. In the regime of weak magnetization, these crossings lead to giant peaks of c refer to the energy and speed of light in the vaccum, respectively. For the reflected light, the electric field readsWe consider a light propagating along mentclass2pt{minimR, T can be determined by using the boundary conditions (Now we introduce a scattering matrix between incoming and outgoing electric fields bynditions . As a re


As an emerging single crystals growth technique, the 2D-material-assisted epitaxy shows excellent advantages in flexible and transferable structure fabrication, dissimilar materials integration, and matter assembly, which offers opportunities for novel optoelectronics and electronics development and opens a pathway for the next-generation integrated system fabrication. Studying and understanding the lattice modulation mechanism in 2D-material-assisted epitaxy could greatly benefit its practical application and further development. In this review, we overview the tremendous experimental and theoretical findings in varied 2D-material-assisted epitaxy. The lattice guidance mechanism and corresponding epitaxial relationship construction strategy in remote epitaxy, van der Waals epitaxy, and quasi van der Waals epitaxy are discussed, respectively. Besides, the possible application scenarios and future development directions of 2D-material-assisted epitaxy are also given. We believe the discussions and perspectives exhibited here could help to provide insight into the essence of the 2D-material-assisted epitaxy and motivate novel structure design and offer solutions to heterogeneous integration via the 2D-material-assisted epitaxy method. Epitaxial growth, a technique to fabricate single-crystalline membranes, has experienced a revolution from the homo- to the hetero-one. In this process, though still governed by the covalent interaction, the application of strain engineering, such as introducing a low-temperature buffer layer or superlattice structure, widens the substrate options to some extent . This so2/MoS2 system as an example, Koma et al. demonstrate the possibility of materials growth on a dangling-bonds-free surface guided by the vdW force. In this scenario, the epilayer is not constrained to the substrate by covalent interaction, thus the epilayer tends to preserve its own lattice structure and not establish the epitaxial relationship with the substrate as that happens in the traditional covalent epitaxy. This growth paradigm offers hope for fully breaking the substrate restriction and is favorable for implementing dissimilar material integration. However, due to the unavoidable presence of dangling bonds on the bulk materials surface, the vdW epitaxy strategy could not be straightly replicated to bulk crystals growth.A layered material growth method proposed in 1984, namely, the van der Waals (vdW) epitaxy, gives inspiration for the further development direction of the bulk material epitaxy . Utilizi2 growth on the epitaxial graphene-covered SiC substrate. The as-obtained heterostructure exhibit ultrafast and efficient interlayer hot electron transfer characteristics and interlayer coupling effects, which shows the potential of being the layered hot electron injecting material [The discovery of layered 2D materials as well as their superior and abundant properties gives a chance to enrich the conventional material set and the fabrication method –6. Ever material . By takimaterial . Besidesmaterial –30. ThouIn this work, we systematically review the lattice modulation mechanisms in 2D-material-assisted epitaxy and give feasible development and application directions of 2D-material-assisted epitaxy. Firstly, based on the concept of lattice transparency and the understanding of the potential relay effect, the lattice modulation mechanism and the possible application of remote epitaxy are discussed. Subsequently, we illustrate the impact of the lattice mismatch in the lattice modulating process in vdW epitaxy. Suggestions for utilizing vdW epitaxy to achieve material growth on arbitrary substrates are also provided. Finally, the possibilities of using quasi vdW epitaxy to manipulate the properties of the epitaxial structure are presented. The discussions and perspectives exhibited in this review may offer help to understand the essence of 2D-material-assisted epitaxy and motivate novel structure and device design via a 2D-material-assisted epitaxy method.2 plasma treatment and ZnO nano-walls growth procedure, a single-crystalline GaN epilayer is successfully obtained and provided a template for the subsequent light emitting diode (LED) structure growth. The as-fabricated LED shows visible blue emission at room temperature and could be transferred onto various target substrates via the mechanical lift-off method. This is regarded as an example of utilizing the optical characteristics of nitrides and mechanical properties of graphene, which demonstrates the potential of the 2D-material-assisted epitaxy for implementing heterogeneous integration [2/Si (100) substrate and enables the reuse of the original SiC substrate. Hong et al. perform the InAs and GaAs nanostructures growth on graphic films in 2011 [Date back in 2010, Chung et al. exhibit the possibility of nitrides growth on the 2D material . Placingegration . Many otegration , 33–37 Kegration . The ste in 2011 . They il in 2011 . Assiste in 2011 . Using a in 2011 .These mentioned-above results exhibit the possibility of 2D-material-assisted epitaxy in cutting the cost of the substrate and achieving dissimilar materials integration via a direct stacking strategy . Along wWith these experimental and theoretical findings, a platform is provided for the subsequent study of the 2D-material-assisted epitaxy. Benefiting from this, the theoretical framework of lattice modulation mechanisms in 2D-material-assisted epitaxy, including the remote epitaxy, vdW epitaxy, and quasi vdW epitaxy, develop rapidly.Many studies have demonstrated bulk materials growth on 2D-material-covered single-crystalline substrates. The obtained epilayers have comparable crystalline quality to the heteroepitaxial ones and exhibit advantages in the exfoliation and transfer process, which greatly satisfies the flexible device fabrication and opens a pathway for heterogeneous integration through the direct stacking method –53. HoweKim et al. first give a systematic explanation based on the concept of “lattice transparency” of graphene and name the modulation mechanism “−6 decay, whereas for the partial ionic bonded GaN, an r−2 decay is expected. As a result, after passing the substrate-graphene gap, the potential of the GaN substrate is still strong enough to guide the epitaxial process, but not for the Si  substrate . The as-er layer . The as-These results indicate that vdW epitaxy is a powerful tool for implementing dissimilar materials integration, which is favorable for the next-generation semiconductor device and integrated circuit development. Besides, the possibility of growing crystals on any growth condition sustaining substrate via vdW epitaxy strategy is also verified, which may benefit the wide usage of large-scale and low-cost substrates in the semiconductor industry . Along wMunshi et al. give an explanation of the lattice modulation mechanism in vdW epitaxy based on the preferred adsorption sites analysis . There aIn vdW epitaxy, the interaction between the epilayer and the 2D material is expected to be a vdW one. Thus, at the initial epitaxial growth process, the demand of creating a covalent bond with a correct bond angle could be relieved and only the adsorptions sites need to be considered. Once the adsorption sites are decided, the subsequent lattice stacking process tends to follow the inherent sequence of the epilayer lattice. However, besides the lack of directivity, vdW interaction is also weak in strength. It is difficult to confine the adsorption of adatoms in vdW epitaxy to exactly predicted favorable adsorption sites, unlike in covalent epitaxy. Therefore, another explanation of the lattice modulation mechanism of vdW epitaxy, which is based on the lattice mismatch perspective, is proposed Fig. c.Ren et al. study the GaN growth behavior on the graphene-covered amorphous quartz glass substrate . Accordi2 and MoS2, are known to be lattice-matched with GaN [2 substrate with the assistance of the mechanical exfoliated WS2 and MoS2 interlayer, respectively [2-assisted vdW epitaxy and study the lattice modulation mechanism inside it. DFT calculations are performed to illustrate the role of WS2 in nitrides vdW epitaxy [2 surface is first investigated. The formation energy results illustrate that N adatoms adsorbing on the top of the W atoms is the only energetically favorable configuration  structure is formed along the growth direction, which results in the as-obtained nitrides lattice of N polarity (Fig. Besides, quasi vdW epitaxy also shows potential in lattice polarity modulation. Liu et al. find that on the N-doped graphene, it is more likely to obtain a N polar nitrides epilayer . After nity Fig. a–e. Whility Fig. f. Experiity Fig. g, h.Fig.These results illustrate that the usage of quasi vdW epitaxy could bring a new pathway to modulate the strain status and dislocation density in the epitaxial structure . BesidesUtilizing the unique and abundant properties of 2D materials, 2D-material-assisted epitaxy gives a chance to revolutionize the conventional epitaxial process. Improving our understanding of the lattice modulation mechanism in 2D-material-assisted epitaxy would have significant benefits for its practical application and future development.Epitaxial growth on the 2D-material-covered crystalline substrate is usually regarded as the remote epitaxy. Ensuring the lattice guidance from the crystalline substrate is the key factor to successfully regulating the lattice of the epilayer in remote epitaxy, which is influenced by the substrate and the 2D material simultaneously. The substrate should have enough ionicity to preserve the strength of the potential after passing through the gap between the substrate and 2D material. This will allow for sufficient modulation of the epilayer lattice. Regarding 2D materials, those with lattice transparency and potential relay effects can maintain the pattern and strength of the potential field from the underlying crystalline substrate. This makes them ideal buffer layers in remote epitaxy. With the well-chosen substrate and 2D material companion, one could achieve transferrable and high-quality single crystal growth via the remote epitaxy strategy. Besides, remote epitaxy also shows potential in modulating the coupling strength at the 3D/2D/3D heterointerface, which offers a pathway to implement dissimilar material integration, even cross-dimensional ones.Another 2D-material-assisted epitaxy strategy, the vdW epitaxy, is a promising candidate for achieving material growth on arbitrary substrates. By selecting the binding nature and the thickness of the 2D material, it is possible to block the potential field of the substrate and construct the epitaxial relationship between the epilayer and the 2D material. To make sure the lattice of the epilayer could be properly modulated by the inherent potential field of the 2D material, the lattice mismatch between the epilayer and the 2D material should be considered. To achieve a satisfactory crystalline orientation through the vdW strategy, it is crucial to have a lattice-matched 2D material buffer layer. This will ensure that the required adsorption pattern is energetically favorable and the first layer atoms could be correctly placed. By matching the lattice of the epilayer with the 2D material, combined with the 2D material transfer process, vdW epitaxy could break the long-standing substrate restriction in the heteroepitaxial growth and offer the hope for achieving the heterogeneous integration via the direct growth method.3 and sp2 complex 2D material surface offers more possibilities for the 2D/3D interface construction process, which thus brings opportunities to design and control the characteristics of the as-obtained epilayer, such as the strain status and lattice polarity, and subsequently offers freedom for the device fabrication.Quasi vdW epitaxy is expected to happen on the artificially engineered 2D material surface. The modified spIn summary, 2D-material-assisted epitaxy provides a fertile playground for creating hybrid systems with unique functionalities and excellent performance. Based on these valuable experimental and theoretical achievements of 2D-material-assisted epitaxy, one could better understand and properly utilize the 2D-material-assisted epitaxy strategy in different application scenarios, which may give inspiration for the design of new type devices and next-generation integrated systems.