We present nucleAIzer, a deep-learning approach aiming toward a truly general method for localizing 2D cell nuclei across a diverse range of assays and light microscopy modalities. We outperform the 739 methods submitted to the 2018 Data Science Bowl on images representing a variety of realistic conditions, some of which were not represented in the training data. The key to our approach is that during training nucleAIzer automatically adapts its nucleus-style model to unseen and unlabeled data using image style transfer to automatically generate augmented training samples. This allows the model to recognize nuclei in new and different experiments efficiently without requiring expert annotations, making deep learning for nucleus segmentation fairly simple and labor free for most biological light microscopy experiments. It can also be used online, integrated into CellProfiler and freely downloaded at A record of this paper’s transparent peer review process is included in the   Microscopy image analysis of single cells can be challenging but also eased and improved. We developed a deep learning method to segment cell nuclei. Our strategy is adapting to unexpected circumstances automatically by synthesizing artificial microscopy images in such a domain as training samples. Identifying nuclei is the starting point for many microscopy-based cellular analyses, which are widespread in biomedical research. Accurate localization of the nucleus is the basis of a variety of quantitative measurements of important cell functions but is also a first step for identifying individual cell borders, which enables a multitude of further analyses. Until recently, the dominant approaches for this task have been based on classic image processing algorithms  organized by Kaggle, Booz Allen Hamilton, and the Broad Institute challenged participants to push the state of the art in nucleus segmentation. The goal of the challenge was to develop fully automated and robust methods effective in a variety of conditions, including differing cell lines, treatments, and types of light microscopy. The challenge attracted thousands of data scientists from around the world. Approaches using deep learning dominated the competition, achieving scores that shattered what was previously possible: the best performing traditional methods we submitted ranked no higher than 1,000 out of 3,891 submissions in stage 1 (data not shown); even classical methods hand-tuned to five subsets of the testing data were beaten by 85 out of 739 submissions in stage 2 testing . The topWe present here a superior approach we named nucleAIzer, which, unlike the previous best submissions, applies image style transfer : an imaghttps://github.com/spreka/biomagdsb). Pre-trained networks for DSB, fluorescent, and histology data can be applied to new images via CellProfiler (https://github.com/CellProfiler/CellProfiler-plugins/blob/master/nucleaizer.py) or through an online interface at www.nucleaizer.org.Our software is open source and freely available . As benchmarks, we include the results reported in the first and second positions of the leaderboard, which was frozen at the close of the competition (https://www.kaggle.com/c/data-science-bowl-2018/leaderboard), a recent deep learning method, unet4nuclei  than the top ranked deep learning approach  on the DSB stage 2 test set and has a simpler architecture with fewer parameters. Our method outperforms all other tested methods, too, including a classical baseline (0.528)  Figure . In addiBBBC039;  .Further information and requests for resources and reagents should be directed to and will be fulfilled by the Lead Contact, Peter Horvath  by Kaggle, Booz Allen Hamilton and the Broad Institute. The competition included a preparatory stage 1, to which teams could submit their solutions during a four-month period and a 4-day long stage 2 final scoring period.Existing nucleus segmentation methods do not generalize well, they perform well only on the limited experimental conditions they are designed or tuned for. The Data Science Bowl was highly successful in the sense that many robust solutions were developed that pushed the state-of-the-art in terms of segmentation performance and insensitivity to image type and quality. Solutions such as ours are now being developed into toolkits for biologists that will accelerate science by improving automation in identifying nuclei.We participated in the competition in both stages, reaching the top 1% in stage1 and top 4% in stage 2. The presented results are based on further improvements post-competition.https://www.kaggle.com/c/data-science-bowl-2018). Our annotated training data included 12 additional data sources besides the DSB data, including some data sources annotated by experts in our institution. This extended the total number of training image/mask pairs from 735 to 1,102, and the number of annotated nuclei from 33,814 to 80,692 (not including the synthetic data). A summary of the data we used is provided in The official dataset for the challenge is composed of a training set and two tests sets, one for each stage. The number of images in each set is 670 (training), 65 (stage 1 test), and 3019 (stage 2 test), stage 1 test masks were released in the second stage. The final evaluation of the teams’ performance was measured on a subset of the stage 2 test set (the identity of the subset remained hidden to the competitors). Many of the competitors used additional data besides the provided training data, as this was permitted as long as participants shared their sources on the official competition website .DSB test1, DSB test2, fluo, and hist. DSB test1 and DSB test2 are heterogeneous test sets from the Kaggle challenge (stage 1 and stage 2). The fluo dataset is fluorescence images of U2OS cells in a chemical screen taken from the Broad Bioimage Benchmark Collection (BBBC039)  . The hishist .https://github.com/spreka/biomagdsb where a detailed documentation can also be found discussing the required versions of frameworks and details about the architecture parameters.Our pipeline is implemented using a shell script to allow continuous execution of the entire pipeline. Python 3 scripts execute the training and inference of Mask R-CNN, U-Net, and pix2pix which rely on the TensorFlow, Keras, and PyTorch environments. The clustering, post-processing, and initial steps of style transfer are implemented in Matlab. Our software is available for download at: postComp method without post processing provides excellent results. For specific experiments with no ground truth annotations, performing the style transfer learning part of our pipeline generates new synthetic training data in the missing domain on which training a new model results in fine segmentation. Alternatively, an online version of our method is available at www.nucleaizer.org.The entire pipeline can be run both under Linux and Windows. In a typical use case, it is not necessary to retrain any of the models. Calling the Our methods were trained and tested on a variety of Nvidia graphics cards, including GTX 1070, 1080Ti, and Titan Xp.We decided to incorporate Mask R-CNN in our pipeline due to its robustness, scalability and instance-awareness. It is currently one of the leading computational architectures in instance segmentation of arbitrary object classes, and its applications dominated the methods submitted to the DSB 2018 competition alongside solutions based on U-Net.U-Net  was specWe applied U-Net in our post-processing pipeline as it can efficiently be used to detect subtle differences such as those around the edges of objects. The network structure is straightforward and computationally feasible.Post-processing the segmented nuclei per se is needed due to the inevitable uncertainty in marginal cases, like relatively small objects most likely corresponding to false detections. We found probability maps predicted by U-Net helpful in such scenarios.preseg) to estimate nuclei sizes as well as to create a mask input for image style transfer learning. Simultaneously, we cluster similar images of the input data into groups, and learn styles on these clusters  is used to make rough estimates about the nucleus size and shape while being robust to changes in imaging modality or magnification. The network is initialized with pretrained weights from the MS-COCO dataset, which contains images and segmentation masks for 91 object types including people, trucks, sheep, dogs, etc. For details about the original COCO competition see lication . The netpreseg, already performed reasonably well on unannotated images in the test set , etc. These groups are used as data sources to learn style transfer models to generate additional synthetic data that mimics the properties of each cluster of unannotated images.To perform the clustering, we use a pairwise similarity metric between feature vectors describing each unannotated image. Features were extracted using CellProfiler  modules https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix). The architecture consists of two adversarial networks, a generator tasked with synthesizing realistic looking images, and a discriminator tasked with identifying real images from synthesized images. This model learns to map one image domain to another through an adversarial loss that encourages the generator to learn to fool the discriminator. The input to the generator in our case is a binary mask containing 1’s at the locations of the desired nuclei, and 0’s elsewhere. The input of the discriminator is an image/mask pair . The generator learns to transform the binary mask into the desired style of the real images from the cluster, and the discriminator encourages this by trying to identify real image/mask pairs from fakes. We use the rough segmentations provided by the preseg network as masks for the unannotated images in the style cluster during learning. We train a pix2pix style transfer network to synthesize realistic images from masks for each of the style clusters.We use the pix2pix  frameworUsing our set of 134 trained style transfer networks, we synthesized 20 new image/mask pairs for each of the styles in the unannotated data. A crucial step for this task was to generate novel binary masks to provide as input to the style transfer network, which uses the mask to generate a realistic image of the cells with nuclei in the locations defined in the mask. We generated the masks algorithmically as a combination of 1) fetching real nuclei masks from a database, and 2) synthesizing nuclei using software simcep; . ApproxiWe assembled our nucleus mask database from images of the official DSB training set and further external datasets see  - some ohttps://github.com/matterport/Mask_RCNN) and wrote handler scripts in Python to create the appropriate data structures and call functions. Training was performed in 3 steps with decreasing learning rate and targeted different layers of the Mask R-CNN network, as described in the documentation of the aforementioned Matterport repository.The synthetic image/mask pairs generated by the style transfer network were added to the annotated training data to update the Mask R-CNN segmentation network. We used the implementation of Matterport  from a ROI of m×m size on the ground truth mask of class k and k that are associated with the k-th class.The loss function was as defined in : it compThe performance of deep learning networks is known to scale with the size of the dataset . TherefoMASK R-CNN is reasonably robust to changes in scale, but superior performance is obtained if the nucleus size is approximately 40 pixels in diameter for the data and parameters we used. preseg nucleus masks such that the resulting mean cell size is set to 40 pixels diameter. Images were then either cropped or padded so that the resulting image was 512 × 512 pixels.Another preprocessing step was to resize the images by a scaling factor to obtain a training dataset homogeneous both in cell and image size. The scaling factors were computed from the size estimation of the The Mask R-CNN model trained as described above is used to predict segmentation masks when new images are provided as input. The images are resized before they are input to the network as described in the previous section.1% were merged, and 3) objects smaller than p2 pixels area were removed. Next, U-Net based correction was performed  a soft margin around the Mask R-CNN contour was defined for each object, with an extension of p4 pixels inwards and p5 outwards. The contour was extended/shrunk based on the U-Net predictions. 3) objects that had in total less than p6 mean U-Net probability were removed. Parameters p1..p6 were optimized on the training set with a genetic algorithm to the DSB-score function (see formulation in section We found that the segmentations could be further improved by postprocessing and refining nucleus contours using U-Net . This enerformed : 1) an ox and the annotation pixels y over the intersection of the two areas. Using a threshold ranging from 0.5 to 0.95 with steps of 0.05, true positive (TP) detections, false positive (FP) detections and false negative (FN) detections were identified. For a threshold of 0.5, a predicted object is considered a “hit” if the IoU is greater than 0.5. For each threshold t, a modified version of precision was calculatedt, TP, FP and FN values as above. We also added a small ε = 10−40 value to the denominators.The evaluation metric used for the DSB competition is based on the mean average precision, as defined on the competition website, at different intersection-over-union (IoU) thresholds. A successful nucleus detection was determined by an IoU test :t, and the mean among the test images. In the following sections, we refer to these measures as mAP (mean average precision), mAR , and mF1 (mean average F1-score).The same strategy was used to calculate mean values for these measures as was for the DSB-score, taking the average over various thresholds We also introduce classification accuracy regarding our style-transfer generated image quality evaluation as follows:NOstyle did not contain style augmented images, AUTOstyle used nuclei masks generated by the preseg network, and GTstyle used hand annotated ground truth to generate nuclei masks. CellProfiler (CP) .Our tests included several variations of our method along with six competing methods and several variations of our approaching using different style augmentation: ler (CP)  is a widilastik . Unet4nuclei  uses a very deep U-Net architecture along with prediction of touching borders. DSB2 also uses a U-Net approach, and forces the network to predict relative locations within each nucleus (https://github.com/jacobkie/2018DSB).We compared against several other approaches including ilastik , which pt4nuclei  is an imlei . In contrast, in our method only a Mask R-CNN and U-Net models are trained for prediction, taking approximately 10 hours for training on the same GPU. The computation time for image style transfer strictly depends on the number of different styles present in the target data as one style model is trained for each, individually taking about 15 minutes. DSB2 uses a simpler architecture.Comparing the complexity as well as the computation time and resources needed to train We also investigated computation time regarding inference with our method. Even though inference time is affected by multiple circumstances including image size, number of objects on the image and VRAM of the GPU used, an approximate one image per 2 seconds can be achieved given the following. An image of 520×696 pixels size having about 120 objects of ~20 pixels median diameter size, rescaled to 2x its original size to have ~40 pixels diameter sized objects, i.e. 1040×1396 pixels resized image, on an Nvidia GTX 1080Ti GPU having 11 GB VRAM can be predicted in 2 seconds.DSB test1, DSB test2, fluo, and hist, described in Section DSB test1, the DSB test1 data was held out from the training set. Similarly, when testing on hist, biomag2 and biomag6 subsets were held out.We tested the methods outlined in Section Methods Used for Comparison on four test datasets: fluo, 21/50 hist, 28/65 DSB test1 images were used for evaluation. None of the final DSB test2 evaluation image set was used for training.The test image sets were used as style transfer learning input as determined by our automatic clustering method: a portion of the set was left out when the clustering algorithm could not find a sufficient number of images for a cluster. Therefore, we report our results on such fractions of the test sets that none of the deep learning networks have seen prior to inference as follows. 100/200 hist, DSB test1 and DSB test2. We also see excellent performance on single domain fluorescence data, fluo. Comparing the results of our method with (AUTOstyle [postComp is the AUTOstyle for DSB stage 2 test] and GTstyle) and without style transfer augmentation (NOstyle), we see a clear trend towards increased performance with style transfer augmentation. If we have access to ground truth nucleus masks (GTstyle) our performance improves, though in many realistic scenarios such masks will not be available. DSB1 and DSB2 are not reported because we did not have access to their code). In The results demonstrate that our style transfer approach improves performance in test sets containing data from heterogeneous sources: preseg and postComp. Note that while postComp was trained on fixed sized (40 pixels diameter) nuclei images and is expected to perform best on objects of approximately the fixed size, preseg is more flexible as we intentionally included images presenting a wide range of object sizes in its training to prepare it for an initial robustness. Therefore we expect preseg to detect objects robustly in a wider size range. We tested both models on DSB stage 2 test set and scaled the images to 0.25–4.0 times relative to our generally expected median 40 pixels diameter objects. Our results confirm our expectations of preseg  which performs significantly better than postComp  on shrunk images as presented on In addition to the qualitative demonstration on We also note that the object sizes can vary on individual images  suggestiWe tested how well our style transfer-generated synthetic images compared to real microscopy images by showing a representative selection of both to field experts (pathologists and biologists) and asked them to tell the synthetic images apart from the real ones. The only prior information forwarded to the participating experts was there are fake images in the collection. Their decision accuracy was measured in a binary fashion: whether the expert could identify a truly synthetic image (1) or not (0). We show an example test image montage below  with theWe report an approximate 57% accuracy (ranging from 42% to 73%) of fake image recognition averaging both our experts and the test cases. Based on the performance of the experts we can conclude the visual quality of the style transfer-generated images is on par with real microscopy images suggesting the advance our approach may bring to cellular compartment segmentation.We visually compare segmentation errors and improvements on We measured such types of errors as follows. 1) a missed nucleus is a false negative (FN) i.e. present on ground truth (GT) with no corresponding object on the prediction. 2) A falsely detected nucleus is a false positive (FP): a predicted object with no corresponding GT. 3) A split nucleus is identified as two or more predicted objects that overlap with a significant region of the best corresponding GT object, respectively; we considered an overlap of at least 30% as significant in this case if two objects contributed to the overlap, and 15% if more. Splits were only considered if the given GT object did not have a single matching predicted object. 4) A merged nucleus is a single predicted object that has a significant overlap with multiple GT objects each. We calculated merges similarly to splits but swapped the role of GT and predicted objects.unet4nuclei produced in total fewer errors than our methods on test set fluo but it has been trained and published on this image set.We conducted our evaluation on the same subsets of each test set discussed in the previous sections. Quantitative analysis of segmentation errors support our results: our method (and its modified versions) generally outperform the compared methods. Comparative results are displayed on CP, ilastik, GVF) tend to predict a higher frequency of false positive objects, typically on complex background regions similar to e.g. Unet4nuclei could not have been trained accurately enough for heterogeneous sets  due to the inevitable uncertainty of U-Net in complex histological regions while it excelled on the single-domain set fluo.Segmentation errors naturally occur in automatic methods. Classical methods  very small or elongated adjacent nuclei with weak textural difference from the dividing background region. Similarly, it unnecessarily split nuclei in cases where texture or edge information may suggest multiple nuclei-like structures inside a single nucleus.Supplementary Table 1Supplementary Table 2Supplementary Figure 8Supplementary Figure 5Supplementary Figure 6Supplementary Figure 7Supplementary Figure 4Supplementary Figure 1Supplementary Figure 2Supplementary Figure 3Data S2Data S1


Mortality following acute myocardial infarction (AMI) has decreased in western countries for decades; however, it remains unknown whether the decrease is distributed equally across the population independently of residential location. This study investigated whether the observed decreasing 28-day mortality following an incident AMI in Denmark from 1987 to 2016 varied geographically at municipality level after accounting for sociodemographic characteristics.A register-based cohort study design was used to investigate 28-day mortality among individuals with an incident AMI. Global spatial autocorrelation (within sub-periods) was analysed at municipality level using Moran's I. Analysis of spatio-temporal autocorrelation before and after adjusting for sociodemographic characteristics was performed using logistic regression and conditional autoregressive models with inference in a Bayesian setting.In total, 368,839 individuals with incident AMI were registered between 1987 and 2016 in Denmark; 128,957 incident AMIs were fatal. The 28-day mortality decreased over time at national level with an odds ratio of 0.788 ) per 5-year period after adjusting for sociodemographic characteristics. The decrease in the 28-day mortality was geographically unequally distributed across the country and in a geographical region in northern Jutland, the 28-day mortality decreased significantly slower (4–12%) than at national level.During the period from 1987 to 2016, the 28-day mortality following an incident AMI decreased substantially in Denmark. However, in a local geographical region, the 28-day mortality decreased significantly slower than in the rest of the country both before and after adjusting for sociodemographic differences. Efforts should be made to keep geographical trend inequalities in the 28-day mortality to a minimum.The online version contains supplementary material available at 10.1186/s12872-022-02519-7. Acute myocardial infarction (AMI) is a serious disease causing many deaths both worldwide and in Denmark. Post-AMI survival has significantly improved in western countries over the last decades largely due to substantial improvements in the prophylaxis and treatment of the disease –5. MajorA strong decline in the post-AMI mortality has been shown from 1984 through 2008 at a national level in Denmark , 8, 9. IThe aim of the present study was to investigate whether the observed decreasing 28-day mortality following an incident AMI in Denmark from 1987 to 2016 varied geographically at municipality level after accounting for sociodemographic characteristics.The present study was a register-based cohort study including adults with an incident AMI  code 410 until end 1993 and 10th version (ICD-10) code I21 from 1994 onwards as primary or secondary diagnosis in the Danish National Patients Register (NPR) or underlying or contributing cause of death registered in the Danish Register of Causes of Death (RCD). All individuals with AMI in the period 1987–2016 were extracted from the NPR and RCD. Furthermore, individuals with previously registered AMI (1977–1986) were excluded in order to include only incident AMI cases in the study period.The study period, 1987–2016, was divided into six 5-year time periods in order to investigate the development over time while still maintaining enough data within each time period. The AMI data were aggregated at municipality level. The average age of the individuals at date of AMI (denoted AMI age) was calculated within each time period and municipality.The validity of the AMI diagnosis in the NPR and RCD has previously been reported to be high . However2), where the healthcare system is tax-financed and access to most primary and secondary healthcare is free-of-charge [The study area consisted of Denmark  who were married or in a registered partnership,“Low education”: proportion of adults with low educational level (“Unemployment”: the proportion of unemployed among adults (unemployed 6 months or more within a year),“Low income”: the proportion of adults in the lowest third of family equalized income ; the cut-off value for the lowest third was based on country level data for every combination of age group and gender).All variables were standardized within each time period by subtracting the average and dividing by the standard deviation. This was done in order to reflect differences between the municipalities rather than temporal trends that may correlate with the 28-day mortality at a national level. Only a low degree of collinearity between the municipality-level sociodemographic characteristics was observed based on Pearson’s correlation coefficients.The municipality-level sociodemographic characteristics were :“CohabitFirst, 28-day mortality in the municipalities was modelled using a simple logistic regression model with time period (mean centred) as the only covariate in order to investigate geographical inequalities in the 28-day mortality adjusting for a national temporal trend (Model 1).Second, Model 1 was extended by including AMI age, and municipality-level sociodemographic characteristics: low education, low income, unemployment, and cohabitation (giving Model 2). Linearity of the covariates was evaluated by visual inspection of parameter estimates of a categorical version of the variables. The residuals for each of the six time periods were mapped (on the odds ratio (OR) scale) to visualize the geographical differences in observed and predicted value after adjusting for sociodemographic characteristics.Data are said to be positively spatially autocorrelated when data points that are geographically closer to one another are more similar than data points further apart. After accounting for sociodemographic characteristics in a logistic regression, spatial structure of the residuals may remain. This is possibly due to an unknown or unmeasured spatially autocorrelated covariate, or because subjects tend to be close to similar subjects (neighbourhood effects) . In ordeThe spatial and temporal structure components reflect the mortality and trend over time, respectively, for each municipality relative to the national mean after adjustment for sociodemographic characteristics. The ORs of the spatial and temporal structure components were mapped to visualize the geographical inequalities after adjusting for sociodemographic characteristics.Model 3 was implemented in a Bayesian setting with inference based on Markov chain Monte Carlo (MCMC) simulations using the function ST.CARlinear from the R-package CARBayesST (version 3.0.1) . The modMoran’s I  was usedData management was performed in SAS software version 9.4 and statistical analyses were done in R version 3.6.1 .From 1987 to 2016, a total of 368,839 individuals with incident AMI were registered out of which 128,957 (35.0%) died between day of incident diagnosis and 28 days after discharge from hospital. An illustration of the construction of the study population is shown in Additional file In all time periods, Moran’s I showed highly significant positive spatial autocorrelation in the residuals of Model 1, which was only adjusted for the time period Table . This inThe residual ORs of Model 2, which was additionally adjusted for sociodemographic characteristics, are visualized in Fig. The estimated spatial and temporal structure component ORs from Model 3 are seen in Fig. The temporal structure component Fig. c displayAfter the spatially autocorrelated random effects were added to Model 3, the spatial autocorrelation in the residuals was further reduced in all time periods compared to Model 2, but significant spatial autocorrelation remained in the residuals in four out of six time periods . A supplementary analysis was performed including only those with an AMI diagnosis registered in the NPR using the same models as in the main analysis . This may be because the included covariates do not sufficiently account for the sociodemographic differences across municipalities. However, it was possible to model all significant geographical inequality in the supplementary analysis by including the random effects (Model 3), which indicated that the spatial autocorrelation in the residuals may be caused by the inclusion of data from the RCD. This could be because the uncertainty of diagnoses only registered in RCD is larger, and the usage of RCD may differ between regions.Identifying geographical patterns in 28-day mortality and the development of these is important in order to identify inequalities in regional health systems and potentially unheeded areas and populations. Previously reported geographical inequality in 28-day mortality was confirmed, and efforts should be made to decrease these inequalities.Based on the results found in the present study we find it important to investigate the possible causes of the relative increase in the 28-day mortality in northern Jutland. Particularly, one could investigate whether the structural changes which were introduced in 2007 had any effect. Moreover, investigating changes over time in the use of general practitioners between different regions in Denmark could be relevant in understanding the spatiotemporal pattern in 28-day mortality found in the present study. Ersbøll et al. (2016) found that number of contacts to general practitioner was associated with AMI mortality . FurtherDuring the period from 1987 to 2016 the 28-day mortality following an incident AMI decreased substantially in Denmark. However, in a local geographical region  the 28-day mortality decreased significantly slower than in the rest of the country both before and after adjusting for sociodemographic differences. Efforts should be made to find the cause of this slow decrease in 28-day mortality and reduce the post-AMI mortality in this region.Additional file 1: Fig. S1. Map of Denmark; Fig. S2. Flow chart.Additional file 2: Table S1. Regression parameter estimates; Table S2. Regression parameter estimates including only individuals with an AMI diagnosis registered in the National Patients Register.Additional file 3: Fig. S1. Proportion of fatal AMI registered in the Danish Register of Causes of Death that were also registered in the National Patients Register; Fig. S2. Residual ORs from Model 2 including only individuals with an AMI diagnosis in the National Patients Register; Table S1. Results of Moran’s I statistic including only individuals with an AMI diagnosis in the National Patients Register; Fig. S3. Estimated spatial and temporal structure component including only individuals with an AMI diagnosis in the National Patients Register.