Objective: A repeatable and reliable follow-up of knee injuries would be desirable to prevent delayed diagnosis and to monitor the efficacy of the applied treatment over time. Ultrasound (US) techniques are an attractive option to this purpose, since they are safe, low-cost and non-invasive. However, its use in the clinical practice is limited by the high dependency on the operator’s experience. Hence, the objective of this study is to provide a standardization of the US image acquisition process for knee osteoarthritis (OA) allowing an extended clinical use of US technologies in this domain. Methods: Clinical specifications were provided by expert musculoskeletal radiologists thus identifying the subject poses and the US probe positions needed to evaluate the cartilage structure, signs of synovitis and joint effusion. Such considerations were used to derive the technical requirements needed for the development of a wearable brace equipped with specific openings to guide the correct placement of the probe. The feasibility of the developed wearable brace was tested on three healthy volunteers, which were asked to acquire informative US images, similar to the reference images performed by the musculoskeletal radiologist. Results: Thanks to the knee brace, the untrained subjects were able to self-acquire informative B-mode images comparable to the corresponding images acquired by an expert clinician. Discussion/Conclusion: The use of a knee brace intended for knee OA US diagnosis demonstrated the possibility to standardize the acquisition protocol and make its application achievable also for untrained subjects, representing a key step toward tele-ultrasonography. I.Osteoarthritis (OA) is a degenerative joint disease that mainly affects cartilage, subchondral bone and synovial tissues et al.et al.Conventional ultrasound (US) B-mode images are widely used in radiology for the diagnosis of several soft tissues i.e., focus, depth, gain and transmission frequency) and the experience of the operator. All these aspects result in a rather high variability and interpretational doubts on the results obtained by different operators Despite such interesting results, US is still affected by some issues that hamper its extensive use in the clinical practice. US imaging is a quite strongly operator- and system-dependent diagnostic modality This study proposes a novel standardized echography acquisition modality for knee OA diagnosis. First, standard poses for the patients and specific positions for the US probe were identified based on the know-how of experienced clinicians in the field and driven by considerations available in the scientific literature A standardized acquisition protocol may pave the way for the use of tele-ultrasonography in OA diagnosis. A remote diagnosis of OA could facilitate frequent evaluations, allowing the early detection of complications and enabling the diagnosis also for people that cannot reach the clinical centers.II.A.A musculoskeletal radiologist with 12 years of experience in musculoskeletal imaging provided specific indications for a complete diagnosis of joint diseases, also according to considerations available in literature In Position 2 see , the subFinally, in Position 3 see  the subj2. To check for the repeatability of the acquisition method, three male subjects  were recruited for collecting reference B-mode images with the procedure described above. All such images were acquired by the same clinician.All US measurements were carried out using an ArtUS EXT-1H system  equipped with a 192 elements linear probe L15-7H40-A5 with a transmission frequency of 15 MHz. The depth was set at 30 mm for the evaluation of the anterior cartilage and 40 mm for the posterior cartilage. The contact area between the probe and the skin was about All subjects gave written informed consent, and the study was conducted in accordance to the institutional ethics guidelines and procedures.B.i.e., location of the probe with respect to the reference systems shown in The probe positions  in which openings were produced in correspondence to the results reported in The silicone inserts were designed according to both the dimension of the probe and the variability of the subjects see  trying tC.To test the usability of the developed wearable knee brace, the same three subjects involved in the US measurements described above were asked to wear the brace and to perform a US self-acquisition, trying to reproduce the corresponding reference clinical images  displayeIII.A.In Based on these data, specific openings were created on the anterior and posterior parts of the brace see . Figure B.The three subjects wore the brace without any problem and were able to reproduce the three positions identified as the optimal ones to image the cartilage , as indiThe B-mode images acquired by the three subjects, shown in IV.US-based knee joint evaluation has advantages over other diagnostic techniques. Indeed, it allows direct, safe and low-cost monitoring of cartilage tissue status and other relevant changes such as the presence of osteophytes, inflammation, synovial thickening, joint effusion, tendons and ligaments and to derive unique information even in the early stage of diseases such as OA In this regard, a wearable brace enabling a guided US imaging acquisition procedure has been designed and developed starting from clinical indications provided by an expert clinician in the field. The clinical guidelines led to identify three poses of the subject in which the cartilage tissue and the possible inflammatory joint status could be evaluated, and OA-like changes identified see . For exaSince the posterior silicon inserts may represent an obstacle when the subject had to bend the knee in Position 1, two versions of the brace were produced: the former enabling the acquisition of the anterior cartilage with the knee bent in Position 1 and 2 see , panel aOverall, the B-mode images taken by untrained subjects wearing the wearable brace  are compThus, the proposed wearable brace proved to be useable as a guidance system intended for US image acquisition also by untrained operators, paving the way for the use of tele-ultrasonography in OA diagnosis.e.g., obstetrics and gynaecology) Although research efforts regarding portable US are quite limited, some interesting results demonstrated the potential of tele-ultrasonography to positively influence the management and treatment in some clinical fields  may acquire US images by using the brace, guided by the reference images previously acquired by the clinician. The collected images will be then forwarded to a specialized clinician and evaluated for the final diagnosis. Therefore, the proposed procedure is not meant to substitute the role of the clinician: the experience of the clinician is always needed to evaluate the results and make the diagnosis. On the other hand, tele-ultrasonography may speed-up the process of diagnosis, allowing more frequent follow-up controls, and providing a non-invasive, fast and repeatable examination of cartilage degeneration and joint inflammation over time, thus improving the quality of patient care.However, the performance of tele-ultrasonography is strictly dependent on the quality of the collected US images: the use of standardized acquisition protocols could significantly help users to obtain accurate and informative US images The correct setting of the acoustic parameters is an important factor, which may affect the quality of US images. For all the positions the acoustic parameters  were preliminary set based on the know-how of expert clinicians. In this study, the same settings were suitable for the three subjects. However, in the future clinical scenario, these parameters could be personalized for each patient in the first examination. Then, each patient would use his/her own pre-setting during the remote follow-up.The correct positioning of the subject is also a crucial aspect, which can influence the acquisition of clinically informative images. In this study, precise guidelines were given by expert clinicians in the field and a goniometer was used to ensure the correct positioning. In the future clinical scenario, alternative solutions could be employed: for example, a dedicated adjustable bench with an integrated goniometer could help the subject to reach and maintain the correct positions in a repeatable way.The authors are aware that further efforts need to be faced to make such a technique effective in the routine clinical practice for OA diagnosis. Future improvements may include the use of embedded sensors to further standardize the probe handling procedures during the acquisition: for example, force sensors may help the control of the pressure exerted on the skin e.g. rheumatoid arthritis.In this study, three subjects with a similar height were analyzed. To assess the usability of the proposed wearable brace on subjects with different heights, two additional subjects were recruited: one much taller  and one much shorter  with respect to the average height considered in this study (181.3 cm). The preliminary results, shown in the supplementary material , suggest that the dimensions and positions of the brace opening are suitable also for such different subject heights. Future efforts may be focused toward making the brace more comfortable and adaptable to subjects of different size, in terms of wearability. For example, the use of an elastic Velcro may allow regulating the diameter of the brace to different knee girths. Future trials will be needed to confirm these results and to further assess the feasibility of such a technique on a broader range of subjects, also including OA patients. Moreover, this approach may be extended in the future to the assessment of other degenerative and inflammatory pathologies, V.In this study, we proposed a wearable tool and a standard protocol for the US examinations of knee injuries. Starting from clinical specifications a wearable brace was developed to guide the correct placements of the US probe to obtain a precise evaluation of the cartilage and other structures of interest. Overall, three positions were recognized to achieve a global assessment of the knee, with particular regards on OA changes and joint inflammation. The feasibility of the proposed wearable brace and the standardized acquisition protocol was tested on three healthy subjects, which were asked to acquire US images similar to the reference images provided by the clinician. The untrained operators were able to acquire informative and correct images in the Position 1, 2 and 3. However, the acquisitions in Position 2 appeared more difficult to manage and thus the subjects required more time to achieve the correct images. Future efforts will focus on the improvement of such standardization method, with the final aim to validate such technology on a wider range of patients. Our results may also open up to the use of tele-ultrasonography, allowing a repeatable, non-invasive, low-cost and fast assessments of the knee joint.•The study has been carried out in compliance with the current applicable ethical-legal framework . Ethics committee approval is not required as it is an observational study.•Participation of volunteers was conditioned to the collection of freely-given informed consent. The information sheet included transparent and exhaustive information regarding the project, partners involved in the observational study, benefits, possible risks, incidental findings procedure, insurance details, information privacy under article 13 GDPR. Only adults and healthy participants have been recruited.•The collected personal data have been pseudonymized and stored in a limited accessed folder. Any re-use of those personal data will be possible only under the conditions stated in articles 5 and 89 GDPR. Authors are committed to specific confidential obligations, according to the “Ethics rules on personal data processing for research and statistics purposes” issued by the Italian Data Protection Authority.Authors declare as follows.


A better knowledge of tree vegetative growth phenology and its relationship to environmental variables is crucial to understanding forest growth dynamics and how climate change may affect it. Less studied than reproductive structures, vegetative growth phenology focuses primarily on the analysis of growing shoots, from buds to leaf fall. In temperate regions, low winter temperatures impose a cessation of vegetative growth shoots and lead to a well-known annual growth cycle pattern for most species. The humid tropics, on the other hand, have less seasonality and contain many more tree species, leading to a diversity of patterns that is still poorly known and understood. The work in this study aims to advance knowledge in this area, focusing specifically on herbarium scans, as herbariums offer the promise of tracking phenology over long periods of time. However, such a study requires a large number of shoots to be able to draw statistically relevant conclusions. We propose to investigate the extent to which the use of deep learning can help detect and type-classify these relatively rare vegetative structures in herbarium collections. Our results demonstrate the relevance of using herbarium data in vegetative phenology research as well as the potential of deep learning approaches for growing shoot detection. The current challenges of climate change require a better understanding of plant growth dynamics, particularly to better measure the evolution of the carbon sequestration capacity of forests . The dynIn temperate climates, the phenology of the fundamental processes underlying tree development and reproduction is coordinated by temperature and light constraints imposed by winter . BroadlyThe great diversity of species present in the tropics is attended by a wide variety of life forms and phenological patterns ,6. Excephttps://www.recolnat.org/en/ accessed on 10 May 2021) provides access to herbarium specimen scans in addition to textual data. These images contain very rich information that, before being exploited, needs to be extracted, for instance through crowdsourcing platforms [The recent and massive digitization of natural history collections, particularly of herbaria, offers new material available for many research activities ,17,18 suficiency , or  to automatically detect a young shoot bearing leaves in extension, (ii) to detect the modalities of growth expression , and even (iii) to eventually distinguish different stages of leaf shoot maturation (phenophases). In the presence of a sufficient amount of detected herbarium samples, and by leveraging their dates of collection, this would then allow for a study of the timing and synchronicity of growth events within populations and communities by analyzing the frequencies of the detected events . With reDo herbaria contain relevant information for the study of tropical tree growth? If so, to what extent?Can deep learning-based automated approaches detect growing specimens?If so, which approaches are most relevant and which growth patterns are best detected?Thus, the main motivation of this work is to try to answer the following questions:http://herbier-guyane.ird.fr/ accessed on 10 May 2021, see A new dataset was created from a selection of specimens located in different ecosystems of the French Guiana territory, scanned and hosted by the Herbarium of French Guiana . Note tThe performance of the global approach emphasizes the need for a more adapted method that takes into account the specificity of the task. In the next section, we study the performance of methods specifically designed for object detection.The performance of the local approaches—Faster R-CNN using bounding boxes and Mask R-CNN using masks—measured by ROC curves are given in In the previous experiments, during training, the models were told if a scan contained one or several growing shoots  and then given their localization in the image . The additional information of the type of growing shoots—“Continuous” or “Rhythmic”—can also be provided and leveraged by the detection models. This is what was studied in experiment EXP3.This study confirms the fact that the proportion of specimens showing new growing shoots is extremely low in herbarium collections. During the preparation of the dataset, the two experts went through 1109 herbarium sheets to certify that 104 pictures (9.46%) contain growing shoots. When we compare this proportion to that of flowering plants on all the specimens of the herbarium on which  have worOur study shows that despite the small number of samples and the size and diversity of visual patterns, automatic detection technologies can nowadays partially predict the presence of growing shoots in herbarium collections. These results are promising and encourage the performance of more investigations and intensive evaluation on larger volumes of training and test data.The performance of the three types of approaches are proportional to the annotation effort in the images: the limited annotations for the presence/absence tag of the growing shoots led to the least good performances, while a time-consuming annotation effort, with masks delimiting the growing shoots to the nearest pixel and a categorization of the type of shoot allowed for the achievement of the best results. Intermediate annotation effort with bounding boxes and a single general category of growing shoot offered an interesting compromise between annotation effort and performance.In the case of a single general category of growing shoots, it seems that the Mask R-CNN approach encounters more difficulties to converge than the Faster R-CNN approach. This observation could be explained by the fact that the mask prediction learning was forced on more heterogeneous visual concepts, which are more difficult to generalize from and would require more data for the model to capture appropriately. Conversely, if the two concepts of growing shoots are considered as two distinct categories, this additional a priori information will be leveraged by the model. In this case, the Mask R-CNN can more easily capture the contours of both types of growing shoots, converge better, and finally outperform the Faster R-CNN.Detecting “Rhythmic” growth is more difficult than detecting “Continuous” growth, especially with the Faster R-CNN model, and to a lesser extent with the mask R-CNN model. Shoots labeled “continuous growth” mostly develop at the ends of branches and are then often arranged on the uniform background of the sheet without overlapping with other parts of the plant. Conversely, rhythmic shoots include lateral branching and can more frequently partially cover or be covered by other parts of the plant. Bounding boxes around a shoot then capture a lot of less relevant content  that probably disturbs the training of the Faster R-CNN model on the rhythmic growing shoots.In the best of cases, when using the Mask R-CNN model, about 87–90% of the herbaria with growing shoots can be found at a rate of 28–30% false alarms according to the current evaluated test set. Furthermore, it is interesting to note that the Mask R-CNN offers the additional possibility of producing masks on which biologists could conduct morphological measurements , bringing more fineness to such types of herbarium-based phenological analyses. However, this approach deserves more investigations on the quality of the masks produced and the risk of overfitting because it seems that predicted masks of better quality are created at the cost of more training iterations, leading to a decrease in the true positive rate see .Deep learning, and AI in general, is a very active research field. The performance of such difficult tasks could certainly be improved in future works, with the latest state-of-the-art techniques, such as with a complementary active learning process . Such meCollect more specimens with growing shoots in addition to those with reproductive structures;Organize the specimens on the herbarium sheet in order to better visualize the ends of the axes, and to avoid leaf overlaps;Annotate the presence or absence of new growing shoots.Our study shows that historical data, with the help of AI technologies, represent an original material with which new knowledge about the vegetative phenology of tropical tree species could be built. However, the following recommendations for the community of taxonomists and biologists who collect and prepare herbarium specimens could further contribute to the facilitation of future phenological studies based on this methodological framework:http://herbier-guyane.ird.fr/ - CAY - accessed on 10 May 2021) of the French national institute for sustainable development (IRD). All digitized specimens of this herbarium are accessible online (http://publish.plantnet-project.org/project/caypub_en accessed in 10 May 2021). Most of them were collected in the tropical rainforests of French Guiana, with the remaining specimens coming from Suriname, Guyana, and Brazil (Amapá and Pará states). The new dedicated dataset consists of a selection of specimens located in different ecosystems of the French Guyana territory [A specific dataset was created from a large corpus already made available to the scientific community by the Herbarium of French Guiana   would have incurred the risk of introducing many isolated and potentially atypical vegetative growing patterns, which would have strongly affected the performances of the automatic detection methods. Furthermore, by excluding these isolated specimens, it was assured that any herbarium containing a growing shoot that would be used for the evaluation of automated approaches would have at least one training example from the same taxonomic group. In the end, the resulting dataset was composed of 408 herbarium specimens belonging to 57 species, 40 genera, and 25 families, with a total of 77 specimens showing the presence of recent growing shoots. These additional annotations of growing vegetative shoots represent the first important contribution of this paper by proposing a new annotated dataset, opening perspectives in phenological analyses based on the observation of this type of botanical structure.During the identification of the growing shoots, we tried to distinguish two categories of shoot extension modalities: “rhythmic” or “continuous” growth. “Rhythmic growth” is manifested by an alternation of periods of rest and periods of active shoot extension or “growth flushes”. A “growth unit” is defined as the portion of an axis that develops during an uninterrupted period of extension . On the “FullImage” level. All herbarium sheets used in this study were associated with a tag (Yes or No) indicating if the sheet had at least one recent growing shoot visible or not. No information on the location, number, or size of recent growing shoots was recorded.“Mask” level. In each herbarium sheet containing one or several recent growing shoots, each shoot was manually annotated and cross-validated by two co-authors using COCO Annotator (https://github.com/jsbroks/coco-annotator accessed on 10 May 2021) a web-based image annotation tool designed to efficiently label images and create training data for object detection tasks . In this study, we first considered a global approach that tried to detect the presence or not of growing shoots in an image without precisely indicating their locations. This first approach was then compared to local approaches that tried to more finely detect the growing shoots through bounding boxes or even masks that locate the vegetative structures at the pixel level. To evaluate the performance of such different approaches, the herbarium dataset was manually annotated at three levels of granularity related to the presence of growing shoots:Before trying to predict the presence of new growing shoots in herbarium specimens, we propose to have a look at the collected annotations to gain insights into this detection task.In Next, we analyze the distribution of the number of growing shoots in the specimens in Finally, we study the surface occupied by each growing shoot in the image to understand the size of the objects to be detected. To summarize this section, we can state that (i) the growing shoots are not randomly positioned in the scans, (ii) they can be numerous in each image, and (iii) they are usually very small relative to the image. These three observations justify looking beyond traditional image classification models, which usually look at the full image scale, and the testing of more local object detection approaches.Detection: What is the performance of automatic growing shoot detection in herbaria? Do we obtain performances comparable to the automatic detection of reproductive structures? With what proportions of missed growing shoots and detection errors?Detection and classification: Is it possible to both detect and classify different types of growing patterns (“Continuous” or “Rhythmic”) automatically?The objective of the experiments is to evaluate deep learning-based automated techniques to detect new growing shoots in herbarium specimens of tropical woody plants. More precisely, the main tasks are the following:global and local approaches. The global approach considers the detection task as a visual classification task and works at the scale of a whole image without specifying which sub-parts belong or not to the different considered categories. In the context of this study, the Detection task is related to a two-category problem (“Without” or with “New Growing Shoot”), and to a three-category problem for the Detection and classification task . The advantage of this approach is that it does not require expensive manual annotation work inside each image of the training set. A local approach, on the other hand, is specifically designed to locate objects, growing shoots in our case, with different levels of precision, such as bounding boxes or fine contour delineations, depending on the power of the method used and the manual annotation effort one is willing to make to train a deep learning model. A total of three architectures were evaluated, one using a global approach and two a local approach, all relying on the same pretrained backbone model , to makGlobal model (ResNet50). The first deep learning model that was trained is the Convolutional Neural Network (CNN) ResNet50 [probabilities) related to the considered categories, but without any information about the sub-parts of the image that contributed to the prediction. Pre-trained models can be easily found for most Deep Learning frameworks, particularly for PyTorch , which was used for our experiments. Details on this model adaptation, data augmentation strategy, and the used hyperparameters are provided in ResNet50 . It was ResNet50  and fineLocal model (Faster R-CNN). The second model that we evaluated is based on the Faster R-CNN architecture [itecture , which witecture . A trainitecture  itself uitecture  as the RLocal model (Mask R-CNN). The third model that we evaluated is based on the Mask R-CNN architecture [itecture , which witecture , using bTo evaluate the ability of the different models to automatically detect growing shoots, the subset of plant observations containing at least one herbarium sheet that showed growing shoots was randomly split at a ratio of 0.66, following a stratified strategy based on taxa. In this way, we reduced the risk of bias by preventing images belonging to the same plant observation from being split between the training and test sets, which would have led to overly optimistic measures of performance. On the other hand, the stratified split by taxa guaranteed that the growing shoots related to one given taxon to be detected in the test set have some training examples related to the same taxon in order to not unfairly penalize the performance. Afterwards, images without growing shoots where added to extend the training and test sets and to introduce “negative” examples that were needed to train the global model EXP2: Detection, Local models EXP3: Detection and Classification, Local models Considering a herbarium collection, a phenological study of the vegetative growth dynamics must be able to rely on images that are deemed certain to contain growing shoots. Nevertheless, it is equally important to ensure with certainty the absence of growing shoots, especially in the case of rhythmic shoots. Indeed, too many false positives, i.e., images wrongly predicted to have growing shoots, could strongly degrade the conclusions of such a study . This is why we retained the main metric of the Receiver Operating Characteristic (ROC) curve. The ROC curve basically gives True Positive Rate (TPR) versus False Positive Rate (FPR) at various probability thresholds. It is typically used in binary classification, as in our experiments EXP1 and EXP2, and it can be extended to multi-label classification with one ROC curve drawn per label (“Continuous” and “Rhythmic” for the experiments EXP3). The Area Under the Receiver Operating Characteristic Curve (ROC AUC) summarizes the ROC information in one number that can be used to select the best model parameters see . The higThe vegetative phenology of plants is an area where much knowledge remains to be discovered, especially for tropical floras. We still do not know the precise growth patterns for the vast majority of species, nor their adaptation behavior in the climatic, geographical, and environmental contexts. These are fundamental research questions that also have an applicative scope in the fields of silviculture and agroforestry, such as knowing the best periods of the year that favor the pruning of trees for better growth and better production of wood, or in the management of trees used for shade in coffee plantations in an agroforestry system.We have shown in this study that herbaria represent a potentially interesting and usable source of information that can serve as material for studies on the vegetative phenology of tropical tree species. However, specimens with growing shoots are quite rare and unbalanced at the taxonomic level since the collection of specimens is often motivated by the presence of reproductive organs. We have shown that deep learning techniques can potentially help to detect at best about 87–90% of the images that truly contain growing shoots but at the cost of a false positive rate of about 28–30%. Future investigations can focus on reducing the false positive rate to ensure the prediction of the absence of growing shoots, which is a crucial piece of information, especially in the case of rhythmic shoots.Automatic techniques can thus be used to enhance past historical data and provide original material based on herbarium specimens of great value for future works on vegetative phenology. Looking towards the future, the study also led us to propose several recommendations for the next collection of herbarium specimens in order to facilitate the implementation of automated methods, and to encourage the consideration of future phenological studies at unprecedented time and space scales that were until now deemed unthinkable by the biology community.


The development of quantitative lateral flow immunoassay test strips involves a lot of research from kit manufacturers’ standpoint. Kit providers need to evaluate multiple parameters, including the location of test regions, sample flow speed, required sample volumes, reaction stability time, etc. A practical visualization tool assisting manufacturers in this process is very much required for the design of more sensitive and reliable quantitative LFIA test strips. In this paper, we present an image-based quantitative evaluation tool determining the practical functionality of fluorescence-labelled LFIA test cartridges. Image processing-based algorithms developed and presented in this paper provide a practical analysis of sample flow rates, reaction stability times of samples under test, and detect any abnormalities in test strips. Evaluation of the algorithm is done with Glycated Hemoglobin (HbA1C) and Vitamin D test cartridges. Practical sample flow progress for HbA1C test cartridges is demonstrated. The reaction stability time of HbA1C test samples is measured to be 12 min, while that of Vitamin D test samples is 24 min. Experimental evaluation of the abnormality detection algorithm is carried out, and sample flow abnormalities are detected with 100% accuracy while membrane irregularities are detected with 96% accuracy. Lateral flow immunoassays (LFIA) form the most widely used point-of-care (PoC) diagnostic tools. The prefabricated LFIA test strips can be employed for qualitative as well as quantitative study of the target analytes . InfectiThe main parameters affecting the performance of lateral flow immunoassays are good reproducibility, accuracy, and sensitivity ,2,5. ManThis paper presents an image-based visualization tool assisting the quantitative LFIA kit manufacturers towards the development of high-performance test cartridges. The developed tool is compatible with fluorescence labelled LFIA cartridges. It provides a practical analysis of sample flow properties, including capillary flow rate, practical location of the test, and control lines. The tool continuously monitors the reaction of the sample analyte at the test and control lines and updates the operator on the reaction progress. It can hence be employed to practically evaluate the optimum reaction time, the feasibility of which is discussed in . There cThe architecture of a typical LFIA test strip is indicated in The hardware architecture of the designed LFIA reader is as indicated in A complementary-metal-oxide type machine-vision camera sensor  with a resolution of 6.4 MPixel was used for capturing the fluorescence. It has a dynamic range greater than 75 dB and is well suited for low-contrast applications. The camera sensor module has an on-chip 12-bit analog-to-digital converter to convert the analogous light intensities to digital images. The captured images are then transferred to a personal computer/tablet through a USB interface. An acquisition system and graphical user interface (GUI) was developed in LabVIEW  to access the camera module. The designed GUI provides the user with the controls of exposure time, gain, and frames to be captured. Materials used for the study include HbA1C and Vitamin D LFIA cartridges , test-specific buffer and conjugate solutions , micropipette , and micro tips.The flow chart of the designed algorithm is depicted in Image data are acquired starting from time t = 0 min to a time ‘T’ minutes as specified by the user. Images are acquired for every 20 s time interval. In low-contrast applications, there is a substantial amount of shot noise present. The current setup involves imaging of static objects alone, and in such applications, we can eliminate the shot noise by averaging over multiple frames, typically greater than two [Once the sample is dispensed onto the sample pad of the test cartridge, it passes through the NC membrane due to the capillary action. Within the initial few minutes of the test, sample flow through the membrane is analyzed. The flow of sample analyte through the membrane is detected through the designed algorithm and verified for any abnormal flow conditions.Flow progress within the NC membrane is determined by segmenting out the flow region of the captured image. Here, the concept of global thresholding is adapted, wherein pixel values of the input image I(t) greater than the set threshold intensity (T) are assigned a binary ‘1’ value, and the rest are mapped to binary ‘0’ as indicated .FM(t) is evaluated by determining the number of pixels in the mid-row of the extracted flow segment as indicated in We track the sample reaction progress at the test and control lines once the sample flow front reaches the sink pad. However, as the sink pad is not within the visible window of the cartridge, the time when the sample reaches the end of the NC membrane is taken as the reference. This is where the sink pad starts, as depicted in For the detection of test and control areas within the segmented NC membrane, the algorithm presented in  is adoptHere, NCM (t) and B(t) indicate the fed NC membrane and the corresponding binary map obtained at a time ‘t’, and  indicates the pixel location and the reaction threshold value set is indicated as RT. The binary map B(t) obtained has to be further processed to appropriately extract the individual test and control area components by verifying the pixel connectivity. Each of the components is labelled and can be independently accessed. To eliminate the dot-pixels segmented as indicated in From the segmented image, as shown in For each input image frame fed every 20 s, the test and control regions are segmented once the sample flow through the membrane is reached. Corresponding AR and VR values are measured for tracking the reaction progress. During the time of the sample flow, the AR and VR values are zero. Later, the AR and VR values initially increase at faster rates, as indicated in For an accurate quantitative analysis of the samples, the LFIA test strip must be free from any irregularities, which could be flow-related issues or some undesirable bright/dark regions in the test and control regions or any constrictions within the NC membrane, etc. The bright/dark regions are observed in the captured image frames due to improperly manufactured test strips or damaged membranes, which could lead to incorrect results. The algorithm was designed to detect and notify the operator of any such irregularities in the test cartridge, providing a qualitative check for cartridge manufacturing.Abnormalities in Sample FlowAs mentioned earlier, sample flow through the NC membrane is due to the capillary effect, which contributes to the uniform flow of sample across the test and control lines, where the antigen–antibody binding occurs. Therefore, the capillary flow plays a critical role in determining the sensitivity. Any obstruction ceasing the flow can be detected by continuously monitoring the flow parameters—flow length, area, and width mentioned in Apart from this, skewing of the sample flow through the membrane, as indicated in 2.Irregularities in Test CartridgeOne of the typical irregularities can be any bright spots present on the NC membrane, apart from the test and control regions, as indicated in 3.Presence of Bright/Dark Regions within Test and Control LinesOnce the test and control regions are segmented, the designed algorithm also checks for any undesired bright or dark regions present within these regions, as indicated in Evaluation of the designed algorithm is carried out with LFIA cartridges from our industrial partner J Mitra & Co. Pvt. Ltd., India. The test cartridge is inserted into the fluorescence-based LFIA reader described in The algorithm initially monitors the flow of the sample through the NC membrane and verifies any abnormalities. Experimental evaluation was carried out for HbA1C test cartridges using whole blood samples. The trend of flow properties, including the flow length, area, and width with increasing time observed for a single test sample for the current set of cartridges employed, is indicated in The algorithm also provides the manufacturer with information on the flow speed of the sample—this aids in the practical evaluation of the capillary flow of the NC membrane material. Once the sample flow is reached, the algorithm performs NC membrane segmentation with the bounding box obtained from the flow measurements as a reference, as mentioned in The NC membrane and the test and control regions were observed to be properly segmented out for all the test samples through the algorithm explained in A similar analysis was done for Vitamin D using serum samples to demonstrate the evaluation for the reaction time algorithm. Image samples were acquired for 30 min. The trend of the VR values and the corresponding slope plots obtained with respect to time is indicated in Once the stability time is determined, calibration of the test concentrations is performed by the kit manufacturers. The VR values measured for the samples under test at the reaction stability time are used to establish the calibration of sample cartridges. The performed calibration is verified by measuring the concentrations of reference HbA1C and Vitamin D samples. The measured VR values are fed to the calibration equations, and corresponding concentrations are determined and compared with the expected values. The experimental study is performed with six HbA1C and three Vitamin D samples. The results obtained for error analysis are indicated in As mentioned in A similar analysis for detecting irreguarities in the NC membrane is done by feeding 120 image frames of proper test strips and 10 frames with bright/dark regions in the NC membrane. The corresponding results are as discussed in The developed algorithm, along with the mentioned hardware setup, forms a portable, simple-to-use evaluation tool, facilitating the research involved in the design and manufacturing of fluorescence-labelled quantitative lateral flow immunoassay test cartridges. The presented algorithm practically analyzes the sample flow rates through the test membrane. The algorithm automatically locates the test and control reaction regions within the captured image frames. The tool efficiently detects the presence of any abnormalities, which could be flow-related or irregularities in the manufactured test membranes. It also helps to determine the reaction time required for the sample reaction, providing a visualization of the reaction progress at the test and control lines. Further experimental evaluation of the algorithm is to be carried out with different sample analytes. During the production run as well, this tool can be used to verify the quality of cartridges. The designed tool can also be adopted in an LFIA fluorescence reader for diagnostic measurements, which determines the quantitative estimate of the samples and detects the presence of any abnormalities in the LFIA test cartridges. This brings down the level of human intervention and thereby provides a reliable, practical solution for increased automation in the field of medical diagnostics.


Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution. It is a widespread assumption that the visually impaired have superior sensory abilities in other sensory modalities. To verify this assumption, many studies have been conducted to show that visually impaired people have superior auditory sensitivity compared with sighted persons . HoweverAll of these systems differ in their methods of converting images into sounds. Among these, vOICe  is one oThe conversion of visual information to auditory signals is relatively limited by the temporal resolution of human hearing. Therefore, it is essential to understand the temporal limitation of auditory sensitivity, which is directly related to the perception of visual information. Traditionally, the evaluation of auditory sensitivity in sensory substitution has been conducted with participants’ feedback. However, human evaluation, in which a number of human participants are involved in a qualitative evaluation, is expensive and time-consuming. Moreover, it is difficult to design an objective method for measuring and analyzing the sensitivity of visual perception owing to significant variations between persons. Because people have different biological and environmental experiences, evaluation results from different participants may vary considerably.Recently, deep learning has emerged as a breakthrough machine learning technique for various areas in computer science, as well as other disciplines. This advancement of deep learning serves as a motivation to exploit artificial perception models as an alternative to human-based evaluation. A recent study  developeThese factors have motivated our proposal of a new experimental method that is both objective and systematic. The purpose of this work is twofold. First, we objectively evaluate and quantify sensitivity in perception of visual-auditory sensory substitution using a machine learning model. Second, we evaluate the extent to which the results of a model-based evaluation of sensitivity analyses correlated with those obtained through human evaluation.To achieve these aims, in this study, we employ a deep learning approach to perception with respect to auditory sensitivity in visual-auditory sensory substitution of the visually impaired. Particularly, we investigate the temporal limitation of sound duration of the soundscape using time-compressed audio signals. To this end, we first conducted preliminary experiments to test whether sound compression could be used to verify the transformation in visual perception from audio sounds in both visually impaired and unimpaired subjects. We then exploited a cross-modal perception model, based on deep learning. Although we took inspiration from the cross-modal learning model, the present work differs in that instead of evaluating the encoding schemes of visual-auditory substitution, we used a computational model to estimate the limit of the auditory sensitivity on usability of the vOICe system. In the cross-modal perception for visual-auditory substitution, the generated images represent the perceived visual information from the conveyed audio stimulus. We attempted to measure auditory sensitivity by analyzing the change in perception of images, which are generated in response to the change in the temporal length of the soundscape provided as an audio stimulation. In addition, we compared the proposed method with an evaluation conducted with human participants to investigate whether these two methods were related. In the human evaluation, we aimed to investigate the auditory sensitivities of blind people by means of a design involving three groups of participants, including sighted users (SU), congenitally blind (CB) users, and late-blind (LB) users. Detailed information on group-specific age and gender are provided in If the temporal length of soundscape is reduced, it is expected to become possible to recognize and respond to rapidly moving objects, because surrounding information can be thus transmitted to the blind more quickly. The hypothesis of our study was rooted in the general assumption that the visually impaired have superior auditory abilities compared to sighted persons. If this assumption is true, we predicted that the latency of the conventional sensory substitution device using a soundscape method, which requires a fixed time to convert an image to an audio signal, could be reduced. We also predicted that there would be a difference in the degree to which different participants  perceived the temporally compressed soundscape sound, and that these phenomena could be confirmed experimentally. We conducted two experiments to test our hypothesis.The remainder of this study is organized as follows: One important consideration for visual-auditory sensory substitution is whether to convey visual information through sound or speech. With the recent development of artificial intelligence and deep learning, there have been several attempts to convey environmental information to the blind through direct audio semantic description. For example, LookTel  detectedIn contrast, most conventional visual-auditory sensory substitution methods transform visual information into auditory representations using a predefined transformation algorithm. The main difference between conventional methods and direct semantic description is that users need to be trained to be able to understand the relationships between the generated virtual sounds and properties of objects in the image. Although these methods may appear to be difficult to use, they can be more flexible for new environments and unseen objects because they produce sounds based on characteristics directly calculated from the input image. Loomis et al.  sought tMany prototypes using a method similar to that followed by  have alrOther approaches that do not use column-wise scanning but rather extract and convert visual information from whole image frames have also been developed. For example, an earlier method, prosthesis for substitution of vision by audition , uses a Recently, an implicit method for visual-auditory sensory substitution technology, called the autoencoded V2A (AEV2A) model , has beeAll of the above-mentioned efforts to develop visual-auditory sensory substitution methods based on cross-modal perception and various psychological studies ,40 have Research on the evaluation of auditory sensitivity for sensory substitution has been conducted in human behavioral analysis. However, this method relies on the subjective feedback of participants; thus, it cannot be considered an objective method. In this section, we describe a proposed cross-modal GAN to analyze auditory sensitivity in visual-auditory substitution and a behavioral assessment to validate it.As described in previous sections, visual-auditory substitution using a column-wise scan has proven useful and their encoding schemes are considered easy to interpret. However, the low conversion rate prevents the visually impaired from taking advantage of these systems in their daily life because it remains difficult to interpret a dynamically changing environment. A solution to this problem may begin with an accurate and objective analysis of the temporal limitation of the visual information that can be conveyed via soundscape. As a basic approach to solving this problem, we use a deep learning model to analyze the temporal limitation of an encoding scheme that can recognize visual information from the auditory signal. In this section, we describe our approach, which is inspired by the work of , to evalOur computational model for auditory sensitivity analysis is similar to that proposed by , in thatThe encoded audio waveform was then transformed into log-mel spectrogram features as the input to the audio encoder. We used a CNN as our audio encoder to model the abstraction in visual perception from translated audio. It consisted of four convolutional blocks and three fully connected layers. Each convolutional block was composed of a convolutional layer with a kernel size of 3 × 3, a rectified linear unit (ReLU) non-linearity layer, followed by a max-pooling layer with 2 × 2 pooling and 2 × 2 stride window, which effectively down-sampled the output of the prior layer. The last layer was used for classification with softmax loss. For the training of the audio encoder model, we used Adam (Adaptive Moment Estimation) as an optimizer with a batch size of 64 samples per step and a learning rate of 0.0001. We trained the system for 10 epochs. After the network was trained, the last layer was removed, and the feature vector of the second to the last layer with a size of 128 was used as the audio embedding in our cross-modal GAN, which was a variant of the auxiliary classifier GAN (AC-GAN) . In the Note that we employed a cross-modal GAN as only part of the experimental method used in this work to evaluate the perceptual variability according to the change in the compression ratio of the audio signal. To achieve our aim, we introduced an additional classifier to evaluate the variation in the accuracy of generated images from the translated sound of the MNIST dataset. For images, CNNs are known to perform well in various tasks. Therefore, we employed a typical CNN architecture  as our bThe purpose of the human evaluation is to expand the experiment by applying the temporal limitation of the auditory sensitivity obtained from the computational model-based assessment. We investigated whether the results of our experiments with human evaluation were consistent with the results of the cross-modal GAN model. We used the MNIST dataset to train a deep learning model for our experiment. However this dataset was limited to 10 categories, and thus was unsuitable for testing sensory substitution, because participants could memorize it. The visually impaired generally prefer to recognize shape first. Therefore, for human evaluation, we used shapes as visual information in order to be faithful to the aim of conversion of images and verification of perception.To examine the effect of the temporal length of the soundscape that could be used for training, we used three different lengths of sounds  for training each group. Reflecting the analytical result that existing studies mainly used soundscapes of two lengths (1 s and 2 s), we used these lengths as normal and long, respectively. In addition, we set the compression ratio for short soundscapes to 50%, considering both the results from the computational model-based analysis, of which more detailed expression will be given below in White images on a black background were conveyed into soundscapes and each duration of soundscape was fixed to one of the predefined times , with the frequency on a log-linear scale from 500 Hz to 5000 Hz. At the beginning of each soundscape, we added a 50 ms click sound to inform the participants of the start of the sound. Each image corresponding to the vOICe sound was printed as a braille shape so that the visually impaired could understand the auditory image by exploring it with their hand. Therefore, as shown in Because most of the participants of the experiment were visually impaired, the overall procedure of the experiment was conducted in a controlled environment, and the visual images used for each training session were programmed to be selected pseudo-randomly. The participants were given no specific pre-training time to familiarize themselves with the conversion rule of the vOICe, but were fully instructed that the auditory sounds which were to be presented were systematically and closely related to the images. They were asked to use and imagine these relations during their training. The training process consisted of five sessions, each session using 15 image shapes. Throughout the training sessions, every visual image was presented simultaneously with its corresponding sound and braille print. The images used in each session consisted of five basic shape images and ten randomly selected images from the transformed shape images. The 15 images selected in this way were used for training repeatedly in a pseudo-randomized order 10 times for a total of 150 training sets per session. Two small groups of visually impaired  belonging to three groups  underwent the training with the encoded sound assigned to their group. The participant of CB belonging to Group A can be denoted as CB A, and trained with the sound converted to 0.5 s. All participants of different groups can be identified in this manner.In this section, we describe the evaluation metrics and analyses of the reliability and validity of the proposed method through experiments. The experiments included a preliminary experiment and a main experiment. Although the authors took initiatives in planning the experiment, defining the problem, presenting a research question, designing the experiment methods, and providing all the materials for the experiment, the participants were recruited and the experiment with the actual test on subjects was performed in cooperation with the Graduate School of Social Welfare, Kangnam University. All the experiments in this study were conducted according to the principles expressed in the Declaration of Helsinki and approved by the Institutional Review Board at the Kangnam University (KNU HR2020010). All the participants provided written and informed consent and received monetary compensation for their involvement. The experimental results are presented and are analyzed below.To improve the efficiency of visual-auditory substitution, it is necessary to properly investigate the extent of temporal limitations that can be conveyed through the replacement of audio signals. Unfortunately, this aspect of the behavioral test has been neglected by most previous studies.A total of 30  participants were recruited into three groups , and each group consisted of 10 people. Age ranged between 20 and 53 years with a mean age of 33.83 and a standard deviation of 10.52. The demographic information of the participants is summarized in We conducted an experiment to evaluate the differences between participant groups in auditory sensitivity by changing the compression ratio of the text-to-sound conversion. The purpose of the preliminary experiment was to explore the baseline auditory sensitivity of visual perception for visual-auditory sensory substitution. To examine the temporal limitation of auditory sensitivity, we defined three types of sentences with different lengths in terms of complexity level. According to the authors of , the immTo obtain more detailed information on auditory sensitivity, the experiment was conducted by increasing the compression ratio from 30% to 70%, which increased linearly by 10%. For audio conversion, each sentence was pronounced by a text-to-speech (TTS) function in response to six predefined compression ratios and the Audition software from Adobe. We performed additional manual work to relieve the crushing of the voice, which was caused by the increase in compression ratio. Thereafter, all the converted audio files were recorded. The sensitivity test consisted of three evaluation sessions corresponding to the complexity level. The experiment progressed in order from short sentences (session 1) to long sentences (session 3). In each evaluation session, participants listened to the TTS-produced sentences using PotPlayer, starting from the sound of the highest compression ratio (70%), to reduce the learning effect through repeated listening.In the case when a participant could not recognize the meaning of a sentence from the compressed sound, the compression ratio decreased, and the experiment continued with the generated sound by adopting the new compression rate. This process of reducing the compression ratio was repeated until the participant recognized the meaning of the sentence from the TTS-produced sound. When the participant recognized clearly the meaning of the sound being played, the compression ratio of that time was recorded as the sensitivity score of the sentence and the experiment was repeated for the next sentence, beginning with the mostly highly compressed sound.During the experiment, we recorded the results and compared them with blindfolded sighted participants performing the same experiments. The data collected for this study were statistically processed using the IBM SPSS Statistics 21 tool for empirical statistical analysis. Frequency analysis and descriptive statistics analysis were performed to find out the characteristics of each participant in the experiment. Among the visually impaired groups, the congenitally blind group showed relatively good performance in the case of the shortest and longest sentences, indicating that subjects were able to extract auditory information with superior sensitivity. This finding confirms the fact that the improvement in auditory performance of visually impaired groups compared to sighted users was indicated by the data recorded. It also suggests that simple text-to-sound conversions can provide a baseline to understand the temporal limitation of auditory sensitivity to visual-auditory sensory substitution.The objective of this section is to investigate the influence of visual perception due to the change in the compression ratio of the vOICe encoding, which is the input for the audio embedding. It is expected that the quality of the generated images from audio embedding worsens as the compression ratio increases. In accordance with this purpose, we first verify the generated images according to the change in the compression ratio of the vOICe encoding. The program for our experiment was implemented using Python 3.5, and the baseline classifier model was implemented using the Keras framework ,52, baseThe inception score is a representative metric used to evaluate GANs for image generation using a standard pre-trained inception network . It compBased on these results, we further investigated the effect of the variation of the compression ratio on the sensitivity for visual perception. As stated in To verify our findings from the measurement of the inception score and FID, we repeated the experiment 10 times. Interestingly, From a practical point of view, it is of interest to test and examine whether the computation model-based evaluation of auditory sensitivity for visual perception is consistent with the results of human evaluation. In this section, we compare our computational model-based methods with the human evaluation to investigate whether the results of these two methods showed similar characteristics.For the human evaluation experiment, we recruited 35 volunteers (15 women) from 20 to 53 years of age with a mean age of 32.4 and a standard deviation of 9.36. The severity grade of all visually impaired participants (S1 to S30) were the most severe level of blindness. The demographic information of the participants is summarized in During the experimental procedure, we tried to ensure that all participants conducted the experiment in equivalent environments. In particular, as described in As shown in Behavioral performance was measured by calculating the proportion of correct answers on each test with a chance level of 0.25. All analyses of behavioral performance were performed with the average scores of tests of participants and groups. The upper row of In our experiment, there was no explanation or prior training of the participants to help them understand the visual-auditory conversion method before proceeding with the training. Therefore, we expected that the change in training time or in visual perception from auditory signals may not appear, or if it appeared, the change would appear very gradually, and that it would be difficult to find meaningful results. However, our experimental results revealed that the process of explaining or pre-training participants was not a prerequisite for sensory substitution. In particular, unlike previous works, an interesting characteristic found in our experiment was that the participants of Group A, who were trained with the encoded sound compressed to half of the normal sound, showed the same change trend as the other groups. This result suggests the possibility that the conversion ratio for visual-auditory substitution of conventional methods, which is the major limitation of the existing vOICe, could be doubled. Moreover, this result also shows that the results of model-based sensitivity analysis and the experimental results of human evaluation are quite consistent.Following the completion of the training sessions, the testing session was conducted. The testing session was conducted as a crossover test to verify whether all participants could recognize the shape even when they listened to the two lengths of sounds from the other groups that they did not hear in the training sessions. In this test, we expected that we could find important clues regarding the training procedure or protocol for sensory substitution.Previous studies have proven the technical feasibility of visual-auditory sensory substitution for visually impaired individuals; however, such methods have remained at the level of laboratory research projects or pilot demonstrations without successful adoption in the blind community because of unsolved practical problems. Therefore, it is necessary to find an efficient approach to improve the efficiency and usability of the existing methods. We were interested in examining whether model-based evaluation could be an alternative to human-based behavioral assessment for sensory substitution. In addition, our main question of interest was whether a deep learning model could be used as an objective method to analyze auditory sensitivity in visual-auditory substitution and whether this model had consistent relationship with human-based behavioral assessment.Our first finding in the preliminary experiment was that the visually impaired generally had superior sensitivity of the auditory sense than sighted users. In general, it is believed that visually impaired people are more sensitive to sound than normal-sighted people; however, many studies have established that visually impaired people are more sensitive to sound than sighted controls for specific tasks. Therefore, we started our work by examining this basic question. Although apparently trivial, the verification of this hypothesis is essential for the efficient development of visual-auditory sensory substitution. Based on the results from our preliminary experiment, we experimentally confirmed that the general assumption about auditory sensitivity of the visually impaired was correct. However, this result remains limited as being found in a preliminary or exploratory experiment, because our experiment was not conducted with a sufficient number of participants.Our most important finding in the examination is that the auditory sensitivity analysis using cross-modal GAN was consistent with that of human behavioral experiments. From a technical point of view, the most difficult and time-consuming task for sensory substitution is conducting continuous experiments with a large number of participants and analyzing their feedback, because most research in this field has generally been clinically and practically conducted. Therefore, this process requires persistence and patience throughout the entire process from the recruitment of participants to the experiment. Furthermore, it is difficult to obtain consistent results, because they depend on the individual characteristics of participants and the experimental environment. In this study, we used a cross-modal GAN model analyzing the auditory sensitivity of the visually impaired to test whether it was consistent with human evaluation. As a result of the COVID-19 pandemic, it was difficult to recruit enough participants who could complete the experiments within a strict time period. Therefore, it was somewhat difficult to suggest a direct correlation between the model-based analysis and human assessment. Nevertheless, it was revealed that the results of our preliminary experiment and the experiment in which the auditory sensitivity obtained from the model was applied to the actual vOICe encoding were substantially consistent.The final finding of this study is related to pre-training. According to an analysis in previous research ,10,11,12Generalization remains an important concern. The training of visual-auditory sensory substitution in this study was provided under a controlled experimental setting. Because we used only two-dimensional simple shapes in our experiment, the findings based on this approach may not be extensible to general objects in a more natural environment containing more complex lines and shapes. Therefore, there are various options for future research in terms of developing systematic training methods and procedures to maximize the effectiveness of learning together with the development of model-based efficient encoding methods.According to the results of our experiment analyzing auditory sensitivity analysis using a cross-modal GAN, the shortest length that could be encoded with vOICe was 0.5 s. We conducted our experiment based on this result. However, we did not conduct further experiment as to whether participants were able to perceive visual information even with a soundscape signal encoded in a shorter temporal length. This could be further extended to a question of whether the shorter latency between visual information acquisition and auditory transduction actually could improve the performance of blind users to perceive a fast-moving object. Therefore, this issue could serve as a future avenue of research that could ultimately lead to the development of more practical aid systems.Another important application of visual-auditory sensory substitution is the perception of depth and distance and the performance of navigational tasks, as is reviewed in Several existing sensory substitution devices have been developed that convert visual information into an auditory or a haptic signal using a predetermined transformation algorithm ,57 or otAlthough our experiment using model-based analysis focused on analyzing auditory sensitivity for visual perception, this approach could be extended beyond sensory substitution to evaluate a more systematic understanding of human perception in wider domain areas such as augmented performance ,61, and In this study, we have proposed a method for analyzing auditory sensitivity based on cross-modal GAN as a basic approach for the efficient development of visual-auditory sensory substitution. To the best of our knowledge, this is the first effort to analyze auditory sensitivity by applying a deep learning model for sensory substitution and to examine the correlation with human evaluation. The results of the computation model-based analysis show that the auditory sensitivity for visual perception was maintained at up to 60% at the compression rate of the vOICe soundscape.Before verifying the results of the model-based analysis, we first conducted a preliminary experiment to explore the baseline auditory sensitivity of visual perception. The results of the experiment showed that the visually impaired had up to a 50% higher auditory sensitivity than sighted users. In addition, we expanded our experiment to verify whether participants perceived visual information from auditory stimuli by applying the compression ratio determined by utilizing the results of model-based analysis to the vOICe encoding. As a result of our experiment, by applying the analysis result of auditory sensitivity using a deep learning model, we were able to reduce the temporal length of the auditory signal for vOICe by 50%. This means that we were able to reduce the latency time for sensory substitution by half that, resulting in a performance improvement of 200%. We also demonstrated that we were able to objectively analyze auditory sensitivity using a deep learning model and the overall process of applying the results of model-based analysis to the encoding of sensory substitution was reasonable, consistent overall and sufficiently feasible.In conclusion, in this study, we have analyzed auditory sensitivity using a deep learning model as a starting point to improve the efficiency of sensory substitution, and evaluated the correlations when the results of the computational model-based analysis were applied to human assessment. While the present work highlights the potential possibility of model-based evaluation for sensory substitution, further research is required to generalize the results of our experiment. Therefore, in our future work, we will aim to answer the fundamental questions that can be asked about visual-auditory sensory substitution, such as, “What kind of visual information should be used for effective learning of sensory substitution?”, “In what order should training proceed?”, “What is the appropriate time for each session of training?”, “How many sessions are required for training?”, and “What is an appropriate time interval between training sessions?” A more comprehensive understanding of these questions would allow us to design a more efficient visual-auditory substitution system, and to develop systematic training protocols for sensory substitution.


This article extends a recent methodological workflow for creating realistic and computationally efficient neuron models whilst capturing essential aspects of single-neuron dynamics. We overcome the intrinsic limitations of the extant optimization methods by proposing an alternative optimization component based on multimodal algorithms. This approach can natively explore a diverse population of neuron model configurations. In contrast to methods that focus on a single global optimum, the multimodal method allows directly obtaining a set of promising solutions for a single but complex multi-feature objective function. The final sparse population of candidate solutions has to be analyzed and evaluated according to the biological plausibility and their objective to the target features by the expert. In order to illustrate the value of this approach, we base our proposal on the optimization of cerebellar granule cell (GrC) models that replicate the essential properties of the biological cell. Our results show the emerging variability of plausible sets of values that this type of neuron can adopt underlying complex spiking characteristics. Also, the set of selected cerebellar GrC models captured spiking dynamics closer to the reference model than the single model obtained with off-the-shelf parameter optimization algorithms used in our previous article. The method hereby proposed represents a valuable strategy for adjusting a varied population of realistic and simplified neuron models. It can be applied to other kinds of neuron models and biological contexts. Large-scale neural network simulations composed of thousands or millions of neurons are useful for better understanding brain information processing primitives. Simplified single-neuron models of low computational cost and based on a few parameters have been proposed to reproduce neuronal firing patterns to encode and decode the information contained in electrophysiological recordings  as the result of the process.Our previous work  or single-objective yet aimed at converging to a single optimal solution. However, the application of multimodal optimizers  and their protocols define the target function that will drive the selection during the optimization procedure.Firstly, the user has to select the particular firing properties of interest of the cell under study extracted from The objective or fitness function, the optimization algorithm parameters and the neuron model parameters  correspond to the initial set-up of the optimization architecture. The multimodal algorithm performs an exploration of the parameter space using a simplified (point-neuron) model template. The neuron models with specific sets of parameters which are obtained during the search  are evaluated according to the objective function. Those model configurations with the lowest total score and different enough from the rest are selected in each iteration and passed to the next optimization iterations during the optimization process. The output of this stage is a sparse population of candidate solutions that correspond to different sets of parameters that stand out in their zone of the search space.Once the algorithm has selected different promising model configurations, the user will validate the most suitable neurons among them. This selection runs in accordance with those neuron models which show biological plausibility of their parameters and reproduce with high realism the firing behavior of the real neuron.In order to demonstrate the potential of the optimization workflow, we applied this methodology to the case of the cerebellar granule cells (GrCs) as a proof-of-concept. This approach allows us to validate the sparse population of candidate solutions obtained and according to the features defined in the existing literature. This section starts by describing the computational neuron model, defining the optimization problem to solve and the experimental pieces of single-cell recordings. It ends with the technical details for simulation reproducibility and data analysis.Since GrCs have a compact electrotonic structure , during current injection (I(t)). Equation (2) describes the evolution of the second state variable, namely adaptation current (w). When the current I(t) drives V beyond the threshold potential (TV), then the exponential term of the slope factor (TΔ) in equation (1) dominates the action potential until V reaches the reset threshold potential (peakV). Then, the reset condition (3) determines that V is instantaneously set to rV and w is increased a fixed amount named b. Both equations (1) and (2) contain 10 free parameters that can be optimized in order to minimize an arbitrary objective function . These parameters are: the total leak conductance (Lg), the leak reversal potential (LE) and the membrane capacitance (mC) in equation (1) that model the passive membrane mechanisms; the parameters TΔ and TV in the exponential term of equation (1) model the spike generation and shape; the time constant parameter (wτ), the subthreshold adaptation (a) and the spike-triggered adaptation parameter (b) define the evolution of the state variable w in equation (2); the parameters peakV and rV that drive the reset condition as mentioned above. These parameters have been set within fixed ranges to constrain the exploring process (initV = LE).Equation (1) describes the evolution of the first state variable, namely membrane potential ( process . The memIn order to obtain a neuron model that replicates the behavior of the cerebellar GrCs we have selected some features which quantify some of the most characteristic firing properties of this neuron type: (1) mean frequency through repetitive firing discharge under direct current stimulation [equation (4)]; (2) latency to the first spike under direct current stimulation [equation (5)]; and (3) burst frequency in response to different sinusoidal current stimulation (stimulation with different oscillation frequencies) [equation (6)]. These features will be combined into a single objective or fitness function to be considered by the selected multimodal evolutionary optimizer.feature_score) related to the spiking features, according to equations . The definition of the objective function that contains all these features is extracted from Marín et al.  between the feature values extracted from the in vitro electrophysiological recordings (expi) and the feature values extracted from the simulated traces from the neuron model (simi). This is multiplied by the weight associated with each feature . The weights of the burst frequency (wBurst frequency) and the mean frequency (wMean frequency) were set to 1 as they both were measured in hertz (Hz) and show values in comparable scales. The weight of the first-spike latency (wFirst–spike latency) was weighted to 1,000 as it was measured in seconds (s). Hence, the algorithm equally weights 1 Hz-error at mean frequency feature, 1 ms-lag at first-spike latency and 1 Hz-error at burst frequency. However, the feature score can be modified (if enhancing the focus on a particular feature with respect to the others) or extended if some extra aspect is to be taken into consideration. For instance, a penalization was used in the definition of the burst frequency score (featurescore Burst frequency) to assure the stability of bursts [as proposed in Marín et al. (The feature score (score function of each feature or objective) is calculated as the absolute distance . These features are extracted from in vitro patch-clamp recordings performed from acute cerebellar slices of a single cerebellar GrC  during 1-s length step-current injections of 10, 16 and 22 pA. The latency to the first spike is defined as the time the neuron takes to elicit its first spike upon current stimulation. Both features are extracted from As introduced, the problem stated in “Model Context and Problem Definition” section has been addressed with a multimodal optimizer, i.e., an optimization algorithm designed to concurrently obtain multiple different global and local solutions to a problem sphere in the search space, and it is treated as an exploration window to center the independent local search component. As an algorithm, UEGO focuses on managing a population of different species, which defines its evolutionary part. It executes the steps shown in Algorithm 1. They are summarized next for the sake of self-completeness, but the interested reader is referred to Jelasity et al.  and OrtiM = 100, N = 10,000,000, r = 0.7, and l = 50 for the present study. The maximum number of species and function evaluations agree with the reference values proposed by Ortigosa et al.  the maximum number of species to keep in the population (M); (2) the maximum number of evaluations of the objective function (N); (3) the minimum radius to keep between different species (r); and (4) the number of search levels or full cycles (l). After preliminary experimentation, these parameters have been set to a et al. , radii bHaving gathered the input, the first step of UEGO is the initialization of its population. For this purpose, it randomly selects a point in the search space, evaluates it, and assigns the first radius to it. By definition, the radius of the first species is equal to the diameter of the search space, which is computed as the Euclidian distance between the lower and upper bounds. Therefore, the region defined by the initial species covers the whole search space, and no solution will be unreachable. After that, the radii assigned to species at creation, and hence the region that they cover, will decrease. They do in geometrical progression with the number of levels until the last one, which is linked to the minimum radius specified by the user see . This stThe second step in Algorithm 1 shows the memetic nature of UEGO. Namely, it consists of launching the local search component. This stage, also seen in step 8, independently affects every species in the population, but at this point there only exists the initial one. As introduced, local search is treated as an isolated component, and the method selected is briefly described at the end of this section. It is only required to start at the center of the species and find a better point in its region after several movements. Theoretically, the local search algorithm is limited by the region of the species, i.e., the radius linked to its starting point. Thus, no single step made by the optimizer in a given species can be larger than the radius. However, every time that the local search component finds a better point in the region, it becomes the new center of that species, so they are countinously moving in the search space. The two previous steps form the first level of search for UEGO. For this reason, the loop in Step 3 counts from 2. Notice that if the number of levels were set to 1 by the user, the algorithm would be mainly equivalent to launching the local search method from a random point. The result would hence be the single species after being locally optimized. Nonetheless, this situation is mostly theoretical. A standard configuration of UEGO is expected to execute several levels of search or cycles. Each of them consists of Steps 4–9.Step 4 defines the computation of the radius that will be assigned to any new species created at the current level. As introduced, they decrease in geometrical progression. This step also involves determining the number of function evaluations that can be consumed for creating and locally optimizing species in Steps 5 and 8, respectively. The budget for creation is always three times the maximum number of species allowed, but that of local optimization increases with the number of levels as summarized above. See Jelasity et al.  and OrtiStep 5 is where UEGO tries to increase its population. It first divides the creation budget among the existing species to calculate how many points will be allowed to evaluate. After that, within the region defined by every species, the algorithm randomly takes the permitted number of candidate solutions. Then, the points of every species are systematically paired with each other, and their middle points are evaluated. If the solution at any of the middle point is worse than that at its extremes, both members of the pair define new species. This is done under the assumption that they are on different sub-areas in the search space. The radius assigned to these new species will be the one that corresponds to the current level, which should be lower than any previous one. By proceeding this way, multiple new species will appear within the limits of every existing one and focusing on smaller regions to concentrate the search. Additionally, notice that UEGO will update the center of the initial species if any of the candidate points considered in their regions is a better solution, so they can move.Step 6 scans the current population to check if the center of any pair of species is closer to each other than the radius of the current level. The goal is to avoid spending too much computing time separately exploring the same region. Species that overlap according to this criterion are fused into a single one. More specifically, the center of the resulting one is that which represents a better solution. The radius will be the largest one of them, which aims to keep the scope of search as broad as possible to avoid premature convergence. Step 7 checks the length of the current population. If there are more species than allowed by the user through parameter M, those with the shortest radius are removed until the population size is in the valid range again. The removal criterion is aligned with the previous idea of maintaining species that allow escaping from low-performing local optima. The last two steps are both procedures already described. Namely, Step 8 will independently launch the local search component from every existing species, which will make them move around the search space. Step 9 rescans the population to identify and fuse any species that overlap after having been moved. The UEGO algorithm ends with step 11 by returning the surviving species. According to the process described, they are expected to be different promising solutions. The separation between them in the search space, i.e., degree of difference, will be the minimum radius defined by the user at least. Therefore, as intended, the users of this method and framework  have sevRegarding the local search component previously referred to, the SASS or Solis and Wets’ method  is denoted as landscape. Using MDS, the differences among landscapes were visualized as distances in the bi-dimensional plane. The input vector of distance to the MDS is calculated as a simple Euclidean distance between landscapes . This method allows studying the different landscapes chosen during the algorithm execution and represents their values in a 10-dimensional space embedded in a 2D plot.To further illustrate the multimodal distribution of the different optimal solutions, we have applied the Classical Multidimensional Scaling (MDS) method  and MATLAB (version 2018b) implementations were used to launch the second stage of the workflow . The proposed pipeline allows simulating the neuron models and calculating the features scores through the Python-NEST environment . For this purpose, through the search, the optimizer manages a population of feasible configurations that are distributed over the search space, can produce new ones, move, and absorb others when they are considered to represent the same parameter set. Thus, the number of candidate solutions that ultimately survive as results for further consideration by an expert might vary. Since the optimizer is stochastic, its results might vary between different executions, so it has been launched 10 times as mentioned in “Technical Details for Reproducibility” section. in vitro reference points). Although they were selected by their well-ranked solutions, some of them (from 18 to 25) have low-performing configurations for some of the considered properties , which means that the algorithm successfully went over the parameter space  showed quantitative fitting to the supra-threshold characteristics defined in the multi-feature objective function with minimized scores. In this section, the top-ranked solutions are qualitatively analyzed regarding their accuracy in capturing this intrinsic excitability of cerebellar GrCs, i.e., firing discharge with a mean frequency increased whereas latency to the first spike firing decreased under injected currents and spiking resonance in the theta range under sinusoidal currents.in vitro measurements contained either one or no spikes. Since there are no points of burst frequencies in higher frequencies in the biological measurements, we have avoided including them in the objective function.Although the top-ranked solutions achieved lower score values, this fact might not imply reproducing the complex firing dynamics of the neuron. The case of the spiking resonance is an appropriate example of this possibility: although the experimental points are suitably adjusted to the graphic curve, the neuron responses are larger when the neuron behavior is extrapolated to higher sinusoidal frequencies. That is, as mentioned in “Feature Measurement” in “Materials and Methods” section, the burst frequencies generated with stimulation frequencies beyond 10 Hz fell to zero because the The whole subset of interesting candidates (from 1 to 12) reproduced the complex spiking behaviors within the limits of our electrophysiological observations. The parameter values obtained for the set of solutions are contained in in vitro cerebellar GrCs , candidate 2 obtained the highest score for the concrete points of the mean frequency feature , within the AdEx models, as we previously hypothesized in Marín et al. . Thus, tSimilarly, candidates 5, 6, 7 and 8 reproduce all the features but with the resonance curves in slightly higher preferred frequencies  the optimization of efficient neuron models that allow the replication of complex dynamics such as the spiking resonance in the theta frequency band while maintaining other typical GrC dynamics such as the regular repetitive firing and the spike timing; and (2) the generation of a diverse population of neuron models with widely explored configurations in sparse local minima. These challenges were addressed by optimizing a multi-feature fitness function defined with the distinctive characteristics of cerebellar GrCs. In this case, the spiking resonance in the theta-frequency band of the GrCs is a complex behavior believed to improve the information processing in the cerebellum . An advantage of using a multimodal optimization method is that it results in a population of candidate solutions which is diverse in terms of parameters values as they best fit the target features at different areas in the parameter space. This allows using the candidate solutions as the substrate for a detailed parameter analysis with respect to the neuron model desired properties. In addition, the algorithm can adjust the parameter exploration according to different working ranges (wide exploration within the parameter boundaries). This population of final candidate solutions characterizes the behavior of the neuronal dynamics across the parameter space, i.e., how neuronal dynamics change as the parameters are modified . The expert is able to match the best parameter set ups towards optimizing one specific feature or another, or rather select a parameter set up to fit all the different target features at the same time (avoiding one feature to dominate against other ones within the combined cost function). The exploration and extraction of a diverse population of solutions facilitate the analysis process of how specific parameters ranges help to adjust particular features . The multimodal optimization, in spite of being a more specific and robust engine , implies a suitable alternative for detailed exploration and analysis of the neuronal dynamics against the single candidate solution obtained by other simpler algorithms which might lose biological information necessary for the subsequent study.The novel workflow presented here constitutes a flexible and versatile tool that can be generally applied to this level of complexity with other commonly used point-neuron models, such as the GLIF or integrate-and-fire neuron models, and with other types of spiking dynamics, as long as the electrophysiological data is available. The multimodal optimization algorithm is only led by the value of the objective function, but this approach does not determine the goodness of the solution (the minimized score), although it is capable of exploring a biodiverse population of solutions according to pre-optimized solutions with interrelated parameters. The pre-optimization allows filtering the solutions according to numerically promising configurations. This facilitates the analysis of the parameter space in relation to the desired neuron properties. The post-optimization is based on the decision of the user. This proposal is an automatization of the population diversity of plausible neuron models for complex spiking behaviors.In our results we already have seen certain biodiversity in the parameter configurations of the final population that can lead to a specific behavior shown by biological cells. If the heterogeneity of GrCs is a real fact in the biology of granule cells, then it could be also reflected in the variability of neuronal dynamics of the neuron models that reproduce the same target behavior . The efficient models and features obtained in this work are mainly to demonstrate the feasibility of the proposed optimization workflow. It can be easily modified by other types of point-neuron models (such as GLIF) or other neuron characteristics in future work. The application to the case of cerebellar GrCs implies taking a step further towards advanced exploration of candidate solutions. It facilitates the evaluation of models based on different neuronal parameters which represent various internal neuronal mechanisms to achieve the target spiking behaviors defined in a complex fitness function.https://github.com/MilagrosMarin/Multimodal-optimization-for-fitting-cerebellar-GrCs.The methodology and datasets presented in this study can be found in an online repository. The repository can be found at: MM and NC: study design. MM, MS-L, and EO: literature and database search. NC and EO: methodology. MM, JG, MS-L, and RC: analysis and interpretation of results. MM, NC, and JG: writing of the article. All the results included in this article are part of MM’s PhD thesis. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.