The eye is generally considered to be the most important sensory organ of humans. Diseases and other degenerative conditions of the eye are therefore of great concern as they affect the function of this vital organ. With proper early diagnosis by experts and with optimal use of medicines and surgical techniques, these diseases or conditions can in many cases be either cured or greatly mitigated. Experts that perform the diagnosis are in high demand and their services are expensive, hence the appropriate identification of the cause of vision problems is either postponed or not done at all such that corrective measures are either not done or done too late. An efficient model to predict eye diseases using machine learning (ML) and ranker-based feature selection (r-FS) methods is therefore proposed which will aid in obtaining a correct diagnosis. The aim of this model is to automatically predict one or more of five common eye diseases namely, Cataracts (CT), Acute Angle-Closure Glaucoma (AACG), Primary Congenital Glaucoma (PCG), Exophthalmos or Bulging Eyes (BE) and Ocular Hypertension (OH). We have used efficient data collection methods, data annotations by professional ophthalmologists, applied five different feature selection methods, two types of data splitting techniques , and applied nine ML methods for the overall prediction approach. While applying ML methods, we have chosen suitable classic ML methods, such as Decision Tree (DT), Random Forest (RF), Naive Bayes (NB), AdaBoost (AB), Logistic Regression (LR), k-Nearest Neighbour (k-NN), Bagging (Bg), Boosting (BS) and Support Vector Machine (SVM). We have performed a symptomatic analysis of the prominent symptoms of each of the five eye diseases. The results of the analysis and comparison between methods are shown separately. While comparing the methods, we have adopted traditional performance indices, such as accuracy, precision, sensitivity, F1-Score, etc. Finally, SVM outperformed other models obtaining the highest accuracy of 99.11% for 10-fold cross-validation and LR obtained 98.58% for the split ratio of 80:20. The eye is considered to be the most important sensory organ for humans and it plays a vital role in the overall ability of humans to interact with the world. Unfortunately many individuals in both rural and urban areas suffer from eye conditions such as cataracts, glaucoma, ocular hypertension, bulgy vision, etc. that affected their vision. There are a variety of causes for this such as age, diabetes, genetic and inheritance. Modern lifestyles, which have led to increased use of displays for digital devices, are also a factor affecting the vision.These eye problems have a particularly high occurrence in many South Asian countries. As an example, in Bangladesh 1.5% of the adult population is blind and 21.6% of the population has low vision according to a survey presented in [We chose individuals assumed to have one of the five eye diseases that are currently prevalent in Bangladesh and collected datafor the diseases (biomarkers and disease symptoms). These data are very important for any ophthalmologist or clinical professional since it is used to guide the treatment of the disease. It can also be used to train artificial intelligence algorithms that can ensure that the correct disease has indeed be diagnosed and provide automated recommendations for how to treat the disease.From a literature search one finds that classic machine learning algorithms, especially classification algorithms, are the most appropriate for the detection and recommendation of the five eye diseases considered in this paper ,3,4.A dataset of eye-disorder-related data was compiled. This dataset will be used to make the eye disease classification easier and and it may also be used for further studies of eye diseases. Multiple machine learning techniques were applied to this dataset to test its applicability to the detection and classification tasks considered in this paper. The dataset can also be used to test other platforms such as Chabot, web apps, etc. which in turn can help the world with more accurate information about eye disease issues. One of the goals in this paper is also to make relevant and accurate medical information about eye diseases available to the medical community.In the case of machine learning algorithms, it is also important that predictions can be explained since this provides insights into the predicted phenomena. Hence the explainable artificial intelligence (XAI) methods used here are of great help for the understanding of the eye disease symptoms for a given disease prediction.Creating a benchmark dataset in the domain of eye diseases validated by professional ophthalmologists, that cam be applied to test ML, AI and Symptomatic analyses.Utilizing ranker-based feature selection methods to identify highly ranked symptoms among the five diseases.Experimenting with scenarios both with and without splitting the dataset and with several feature selection methods for better predictions.Compare the performance of classic ML methods to efficiently predict the occurrence of eye diseases.The main contributions of this study are:For the remainder of the paper, In this section, a brief overview of the five eye diseases that we have worked on is presented. For each disease, we describe the disease conditions, disease symptoms and disease risk factors. The images of the selected five categories of eye diseases are shown in A dense and cloudy area that is usually formed on the lens of the eye is called a cataract. This cataract is an agglomeration of proteins situated in the eye that forms a lump-like mass which prevent the normal transmission of light through the lens to the retina. Some of the major symptoms appearing due to cataracts include degradation of night vision, blurry vision, faded colors and double vision. The different sub-types of cataracts are nuclear, cortical, posterior, congenital, senile, traumatic, bilateral as well as other subtypes. These sub-types are grouped into one disease category for this study to minimize the complexity of asociating the cataract with a specific sub-type. Smoking, older age, obesity, increased blood pressure, history of the same disease in family members, diabetes, exposure to radiation from X-ray and cancer treatments are the main risk factors associated with cataracts . CataracAACG is caused by a sudden increase in the intra-ocular pressure (IOP) which causes a displacement of the apposition of the iris to the trabecular meshwork. In acute angle-closure (AAC), the common symptoms are blurred vision, red eye, pain, headache, nausea and vomiting . The proPCG is another type of glaucoma caused by damage to the optic nerve . High inBulging eyes (BE) which is also known as exophthalmos, the medical term for BE. It is not a condition by itself, but a symptom of other conditions. Symptoms of BE include excessive dryness, visible whiteness, double vision and throbbing sensation in the eyes . One or Ocular hypertension (OH) is caused by poor drainage of the aqueous humor, the fluid inside the eye. Risk factors for developing OH are patients having a previous family history of ocular hypertension, glaucoma, diabetes, and age over 40. Patients having unbalanced diabetes or high blood pressure may be exposed to ocular hypertension .Papers relating to the prediction of eye diseases using artificial intelligence and machine learning are reviewed in this section since these areas discuss the tools that are used for the proposed method for efficiently predicting the five eye diseases discussed above. Some papers discussing explainable artificial intelligence methods are also included since some results from these paper are used for better interpretation of the results from the proposed method.The few works that have been performed in the area of eye disease prediction using symptomatic data are also considered here.One study assessed the prevalence of eye diseases in a certain low-income community in Bangladesh. The study was performed using in-person surveys and the results showed a higher prevalence of eye diseases than the world average. This reflects the difference in eye disease prevalence and diagnosis between individuals in low-income individuals and individuals with higher standards of living . The stuRef.  is an epKadir et al.  performeWe found some articles presenting methodologies to address eye disease classification using image processing techniques and ML algorithms. For these methodologies it is important for classification to have a large enough set of image data to run the models with image processing abilities that detect the region of interest (ROI) in the image. One article, Sakri et al. , presentNeural network based approach using the image dataset is a very common scenario in detecting eye diseases. Nazir et al. , extractA data-driven approach for eye disease classification was adopted by Malik et al. . The autA recent work  presenteFrom the review of the existing works, we found a gap in finding the most relevant symptoms for different types of eye diseases as well as the lack of a benchmark dataset with symptomatic data of eye diseases. In this study, we have focused on these gaps and we have designed a research methodology to fill the gaps.An overview of the research methodology adopted for eye disease predictions in this study is shown diagrammatically in Data collection processes have in general to be executed rigorously in order to create a valid and useful dataset. The data collected has to include both symptoms and biomarker data for each patient included in the dataset. For this study, we have included real-world patients data collected when they were present during a referral to an ophthalmologist. Since it can be very difficult to track the symptoms by the ophthalmologist during a short visit, there were two interviewers  collecting the data. The statistical properties of the dataset acquired are presented in The patients were examined by a practicing ophthalmologist while checking the conditions and biomarkers listed in Data annotation or labeling of the dataset used for the training of a methodology that is based on supervised machine learning algorithms is the most critical and important step in the development of the methodology. If the annotation or labeling of the data is not performed properly then the acceptability of the whole methodology might be questioned. In this case, the data collection was performed very carefully. While collecting the data, the ophthalmologist played the role of an interviewer and the patients played the role of an interviewee. Questions associated with the 19 attributes were asked and properly examined by the ophthalmologist to determine the right kind of eye disease that the patient had. Hence, the ophthalmologist worked as the annotator of the dataset. Therefore, the dataset created for the study has been validated by domain experts, hence it can be considered as a benchmark dataset in this domain. Each attribute is labeled either zero (0) or one (1) based on the examination performed by the ophthalmologist. The annotated dataset contains 19 attributes/symptoms and one (1) class label indicating the eye disease. This annotated dataset was forwarded to the next step in the feature selection process.For the feature selection step, we considered ranker-based feature selection methods, so that we got a list of attributes, in this case, symptoms. It is expected that this selected symptom list may be significant for predicting a particular eye disease more efficiently.The ranker-based methods utilized in this study are Pearson Correlation (PC), Information Gain (IG), Principal Component Analysis (PCA), Relief-based Ranking (RR) and all features. The methods were applied directly to the annotated data and the ranking scores of each attribute were determined. Brief descriptions of the feature selection methods are given next.X and Y and X and Y:The Pearson Correlation (PC)-based feature selection  method iEquation  where coX and Y can be considered as class-feature or feature-feature relationships. X can be the class and Y can be the feature or X can be the feature and Y can be another feature. The X and Y where 0 means no correlation between X and Y and +1 means a positive correlation between X and Y. The higher the X and Y is. Therefore, for our study, we have chosen the class-feature relationship. We calculated the class-feature correlation values for all the features and ranked them by the correlation values from high to low. Depending on the ranking, we selected some of the features to run further steps of the ML models.The Information Gain (IG) is another widely accepted feature selection method for various research problems for example where text categorization is used ,32. ThisH is the entropy, and x for Here, Principal Component Analysis (PCA) was invented by Karl Pearson, see ,35, initp instances of n number of features. The method iterated m times starting with a n-long weight vector W. In each iteration the weight vector is updated as in Equation Wi=WX) is assigned to one random instance, and the closest same-class instance is called ‘near-hit’. Similarly, the closest different-class instance is called ‘near-miss’. Therefore, the weight of any given attribute will decrease if it differs from that feature and increase in the reverse case.For each iteration, the feature vector  was implemented with the feature selection methods. The method may choose a specific number of features to be retained for the remainder of the process. We have kept the default value -Test(%) technique, we split the data using 66–34%, 75–25% and 80–20%. Using the different splits we ran the same models on the data, to find the performance of the ML algorithm.k equal-sized sub-samples. Among those k sub-samples, one sub-sample was considered for testing and the rest of the sub-samples were considered for training and the same process was run k-times. In the same manner, as for the Train-Test strategy, we chose 3-fold, 5-fold and 10-fold Cross Validation for finding the best-performing ML algorithm.The second data splitting strategy is k-fold Cross Validation . Though The naive Bayes classifier simplifies the classifying process considerably by assuming that the presence of a particular feature in a class is not related to any other feature in the class . AlthougThe k Nearest Neighbor (k-NN) method is a popular classification method in data mining and statistics because of its simple implementation and excellent classification performance . HoweverFor a simplified description, decision tree analysis is a divide-and-conquer approach to classification (and regression which is not covered within the scope of this review) . DecisioIn the same manner, as contingency table analyses and two tests, Logistic Regression (LG) allows the analysis of dichotomous or binary outcomes with two mutually exclusive levels and it allows the use of continuous or categorical predictors and provides the means for adjusting for multiple predictors . This maWe adapted the widely accepted measurement tools accuracy, precision, recall and F1-score for evaluating the performance of the applied ML methods. Usage of these measurement indices can be found in many existing works, including ,46. WhilHere, In this section, we show the detailed results of the experiments. After applying the feature selection methods the selected features along with their ranking values are given in this section. The experiments were executed based on splitting and feature selection. The performance measurement indices mentioned in We applied multiple ranker-based feature selection techniques and the ranking score of the features is therefore important when choosing the high-scored features for the further process. The ranking score, attribute names and numbers are shown for PC, IG and RR in We devised four (4) experiments to test multiple data-splitting strategies and feature selection techniques. All of the ML methods were applied in these experiments. For the first two experiments, we considered splitting with and without the feature selection applied. For the next two experiments, cross-validation was applied with and without feature selection methods. As described in Data splitting was performed and five feature selections were applied for this experiment. The effect of the selection features can be found in this experiment. Comparison of precision and recall/sensitivity values are shown in The accuracy values for each of the models for different train-test split scenarios are shown in For experiment-2, splitting was applied, but the FS methods were not applied. Precision, recall, F1-score and accuracy of the ML models are shown in In experiment-3, we applied a cross-validation technique instead of a split with the application of five feature selection strategies. This resulted in a total of one hundred and thirty-five (135) runs of the ML models with different setups. This rigorous experiment gave us the most suitable ML model among all the models chosen for the study. The precision, recall, F1-score and accuracy values are presented in In experiment-4, we applied three different cross-validation methods, but this time without selecting any particular features coming from the FS methods. This method can be easily compared to experiment-2 to compare the percentage split and cross-validation methods. The list of significant features based on the feature selection algorithms is depicted in Considering the selected feature sets in Equations –11), th, th11), Considering In this section, we have compared our proposed work with the existing works on the same domain focusing on the prediction of eye disease. Most of the works from literature, used image data to detect eye disease. The images contain the disease region of interest (RoI) and it is easier to identify the RoI by the experienced ophthalmologists. The challenge of annotation can be handled easily by the domain experts and then the supervised learning algorithms would be appropriate for the prediction. The recent advancements of machine learning and deep learning methods also encouraged the researchers to exploit them for the eye disease detection. The following This paper presents an efficient approach for predicting five different eye diseases and shows a comparative analysis among the ML methods for predicting the five common eye diseases from a benchmark dataset. The most critical issue with suitable dataset for ML models is annotation or class labelling. For our work, we have annotated the data by practicing ophthalmologist, which gives more accurate and validate data for the models. Therefore, the ML models are showing very satisfactory results in terms of accuracy values, as most of then showing accuracy above 90% and all the models are showing accuracy over 70%. The highest accuracy obtained is 99.11% from SVM with cross-validation and without applying any feature selection methods. The significant features are identified using feature selection methods and the intersection of the selected features are showing the common features. From the common features we obtain the understanding of the symptoms responsible for the eye diseases. One of the shortcomings of the paper is that we have not used any images for the predictive analysis. Using image data along with the annotated symptom data would have provide better solution to a multi-model approach. In future works, a multivariate or uni-variate analysis may be conducted to identify specific symptoms and acquire insights about a particular eye disease. The application of explainable artificial intelligence to interpret the best model could be another improvement of this work.


With the development of information transmission, there is an increasing demand for state monitoring of fiber-optic communication networks to improve the security and self-healing ability of the network. Distributed optical fiber sensing is one of the most attractive methods because it can achieve real-time detection of the whole network without additional sensing heads. However, when the sensing network is introduced into the communication network, the failure probability should be efficiently suppressed with limited sensing resources. In this paper, the fuzzy logic system is used to evaluate the impact of different sensing resource allocation on optical cable network quality. The link failure probability and path failure probability under the condition of uniform and non-uniform sensing resource allocation are simulated and analyzed, respectively. As shown in the analysis results, the failure probability under non-uniform allocation is significantly lower than under uniform allocation. In this paper, we discussed and addressed the allocation of the optical fiber sensing and communication integrated (OFSCI) network with the limited sensing resource for the first time. The results are helpful to develop an allocation strategy for optical fiber sensing and a communication integrated network with a higher robustness. Optical fiber communication plays an important role in many fields, such as electrical power systems, environmental monitoring, infrastructure construction, etc. Therefore, the long-term security state of optical fiber networks is the basis to ensure the normal operation of the operation system. Detecting or predicting the security state of the whole network is an intuitive and efficient strategy. Distributed Optical Fiber Sensing (DOFS) is one of the most attractive methods, which can achieve physical state monitoring of the whole optical fiber network in real time. Besides, DOFS has the advantages of simple installation, low cost, high reliability ,7,8,9,10However, the OFSCI network is more sensitive to path failure than an ordinary fiber sensing network because in OFSCI the fiber sensing signal shares communication paths with other communication signals, instead of using its independent paths ,18,19,20In this paper, we discussed and addressed the allocation of the optical fiber sensing and communication integrated (OFSCI) network with the limited sensing resource for the first time. We used the fuzzy logic system to assess failure probability of paths and links. The fuzzy logic system is an effective method to solve the contradiction between accuracy and validity . FirstlyThe fuzzy logic system mainly includes four parts, which are fuzzifier, fuzzy rules, inference machine, and defuzzifier. The schematic diagram of the fuzzy logic system is shown in The fuzzifier is a mapping of setting real value points (crisp input) to fuzzy input sets by membership function. The value of membership function is between 0 and 1 on the closed interval, which is used to represent the corresponding degree of input variable to the fuzzy sets . The fuzFuzzy logic systems have Single-valued fuzzification refers to: Map the crisp input Each fuzzy rule represents a fuzzy implication relation. With product inference machine, assuming T both indicate the chosen t-norm. With center average defuzzification, crisp output is expressed as:l-th output fuzzy set. The degree of firing level corresponding to the The degree of each node is greater than or equal to 3 in the network topology, which consists of 20 communication nodes, 40 links. In order to find the transmission path with the lowest failure probability between any two communication nodes in the network, a fuzzy logic system can be utilized. Note that the failure probabilities are determined by sensing points, link lengths, link orders, and data volumes.The four input variables  are shown in For uniform allocation mode, 5000 sensing points are equidistant distributed on each link, as shown in For non-uniform allocation mode, the sensing points of each link are shown in The failure probability of each link under the two allocation modes are shown in i and node j is connected, i, first find a node j with k with j. Then, repeat this operation until the end node, and record the path. In this connected network structure, more than 7000 paths between node 1 and node 6 are obtained by traversing the adjacency matrix A. For any two communication nodes, there is only one link, but there can be multiple paths through different links. In our simulation, the path failure probability is calculated by a fuzzy logic system combined with duality theory. In this subsection, a pair of nodes with sequence numbers 1 and 6 in According to the link failure probability output by the fuzzy logic system in two modes in Furthermore, the path failure probability of 20 communication nodes is simulated, and the lowest failure probability of any 2 nodes. The results of the lowest failure probabilities are shown in In this paper, the allocation of limited sensing resource in optical fiber sensing and in a communication integrated (OFSCI) network is discussed for the first time. We use the fuzzy logic system to assist sensing resource allocation for the OFSCI network. The link failure probability and path failure probability based on fuzzy logic system are simulated, respectively. And the results of uniform and non-uniform allocation with limited sensing resources are analyzed. The experimental results show that the failure probability with non-uniform allocation is lower than that with the uniform allocation. As a result, the highest value of the simulation results of the lowest failure probability between any two nodes is decreased from 0.21 to 0.13 under non-uniform allocation, in the network consists of 20 communication nodes. The method proposed in this paper can reasonably allocate the sensing resources and further improve the transmission robustness of the optical fiber sensing and communication integrated network. In the future, we may study the adaptive fuzzy logic system to realize the optimal sensing resource allocation for the OFSCI network. The adaptive fuzzy logic system is realized by combining a neural network and a fuzzy logic system. The neural network can make full use of model information and expert knowledge, which can effectively improve the automatic control ability of the fuzzy logic system.


Complex two-dimensional warranty equipment is usually composed of many multi-component systems, which include several key components. During the warranty period, conducting maintenance according to the preventive maintenance plan of each component will increase the warranty costs. Opportunistic maintenance is an effective approach to combine the preventive maintenance of each individual component, which can reduce the warranty cost and improve the system availability. This study explored the optimal opportunistic maintenance scheme of multi-component systems. Firstly, the failure rate model and reliability evaluation model of the multi-component system considering failure dependence were established. Secondly, the preventive maintenance plan of each individual component was determined, with the goal of obtaining the lowest warranty cost per unit time in the component life cycle. Thirdly, the preventive maintenance work of each individual component was combined, and the two-dimensional warranty cost model of the multi-component system was established according to the reliability threshold when performing opportunistic maintenance. In the experimental verification and result analysis, the genetic algorithm was used to find the optimal opportunistic maintenance scheme for the power transmission device. The comparative analysis results show that the opportunistic maintenance scheme reduced the warranty cost by 5.5% and improved the availability by 10%, which fully verified the effectiveness of the opportunistic maintenance strategy. Warranty refers to the maintenance conducted by the manufacturer after equipment has been put into use . With thNew complex equipment is usually composed of many multi-component systems, and warranty activities are usually carried out for individual components constituting multi-component systems. Two-dimensional warranty strategies are widely adopted in new complex equipment, in which one dimension represents calendar time and one dimension represents usage degree (such as driving mileage or usage time). When the warranty period of any dimension reaches the predetermined period, the warranty ends . For exaMaking reasonable two-dimensional warranty decisions for complex equipment is a common concern of manufacturers and users. Two-dimensional warranty decisions mainly refer to determining a reasonable preventive maintenance scheme to reduce costs within a warranty period. Due to the different failure laws of components constituting a multi-component system, the isolated maintenance of each individual component will lead to excess maintenance and increase the frequency and cost of preventive maintenance. On the other hand, mechanically combining the preventive maintenance work of various components will lead to the advance of the overall preventive maintenance work and waste the use value of equipment . Most ofThis study adopts the opportunistic maintenance strategy and two-dimensional warranty method to determine the replacement cycle of each individual component in the component life cycle with the goal of achieving the lowest warranty cost per unit time.Through opportunistic maintenance, the preventive maintenance work of each individual component is adjusted according to the reliability threshold when carrying out opportunistic maintenance. Then, the opportunistic maintenance plan of the multi-component system is formed.In a case study, a power transmission device is used as an example for analysis. The reliability threshold for opportunistic maintenance of each component is solved using a genetic algorithm. Finally, the effectiveness of the proposed method is verified through comparative analysis.On the basis of existing research, this study expands the one-dimensional warranty to the two-dimensional warranty, and considers the failure dependence between multiple components, thus, broadening the scope of application of the model. The main problems solved in this paper are detailed as follows:This article is organized into seven sections. Before making warranty service decisions for a system, the system characteristics should first be analyzed, then the warranty strategy should be determined, and finally, the maintenance strategy should be determined. This paper mainly focuses on the failure dependence of multi-component systems, adopting a two-dimensional warranty strategy, as well as an opportunistic maintenance strategy. Therefore, the literature review was mainly performed in three parts: A “system” usually refers to an entity with certain structures and functions composed of a certain number of components or subsystems . System Warranty services are contractual relationships established between the manufacturer and the consumer (or user). If a product fails within the warranty period, the manufacturer is responsible for rectifying the product failure. Warranty services can not only guarantee the product performance, but also remove psychological worries of consumers about uncertain product quality. From the perspective of manufacturers, warranty services can not only improve consumers’ satisfaction and recognition of products, but also establish brand image and bring potential new customers; from the perspective of consumers, warranty services can not only protect their own rights and interests, but also reduce the product maintenance costs during the warranty period. Warranty strategies are key indicators to measure the quality of warranty services and are also important factors which influence consumers’ purchase decisions. The feature of two-dimensional warranties is that the warranty period includes two variables: calendar time and usage, which can usually be represented by an area on the two-dimensional plane. The warranty policy can change the shape of the warranty area. Wang  studied n, N) opportunistic maintenance strategy suitable for a two-component series system. The  strategy means that when the running time of a component reaches N, preventive replacement is performed. At the same time, it is judged whether the running time of other components reaches n; if so, opportunity replacement should be conducted. Zheng et al. [Opportunistic maintenance refers to the maintenance activities performed for the maintenance of one component bringing opportunities to perform the preventive maintenance of other components, to carry out some preventive maintenance work in advance. Berg  studied g et al.  proposedg et al.  studied g et al.  and otheg et al.  and otheg et al.  and otheIn this paper, the reliability index of each component is regarded as the basis of maintenance, and the reliability threshold when carrying out preventive maintenance and the reliability threshold when carrying out opportunistic maintenance are set, corresponding to the condition-based maintenance strategy and opportunistic maintenance strategy, respectively. Firstly, the preventive maintenance plan of each individual component is determined by the condition-based maintenance strategy; then, the preventive maintenance plan of each individual component is adjusted and combined with the opportunistic maintenance strategy. This study took the series multi-component system as the research object. During the two-dimensional warranty period (1)The initial reliability of the component is 1, i.e., it is put into use from new.(2)Maintenance resources are sufficient, and the situation of waiting for maintenance due to insufficient maintenance resources is not considered.(3)There is only one main failure mode of components; multiple failure modes are not considered.(4)The utilization rate can be obtained from historical data, and the usage habits of the same user remain unchanged during the basic warranty period, i.e., the utilization rate remains unchanged, and the utilization rates of different users obey uniform distribution.(5)When component (6)In cases of failure during the preventive maintenance interval, the corrective maintenance (minimum maintenance) shall be adopted for component S components in total. The direct construction approach in the univariate method was used to construct the two-dimensional failure rate model of a single component. Based on the construction method developed by Yun and Kang [The multi-component system contained and Kang , the faiStep 1: The component sample size is determined as Z (the number of samples used for the experiment was Z). ith  represents the warranty cost per unit time during the component life cycle, which is mainly composed of four parts. Here, By solving Equation (9), the optimal values of There have been many studies conducted on two-dimensional warranties. During the two-dimensional warranty period, there are many maintenance strategies available. For example, Iskandar and Murthy  divided In this study, the opportunistic maintenance strategy means that when preventive maintenance is carried out for a component at a certain time, multiple components with similar preventive maintenance times are subjected to preventive maintenance as well. Thus, opportunistic maintenance adjusts the maintenance time for certain components and leads to early component maintenance. It is common knowledge that performing maintenance ahead of schedule will reduce the failure risk of components and reduce the maintenance costs caused by component failure, although it will waste a certain use value of components. Therefore, it is necessary to control certain conditions when carrying out opportunistic maintenance. Thus, the reliability threshold when conducting opportunistic maintenance for components is introduced.t. If it is less than i is not considered. Comprehensively considering reducing expected failure maintenance costs and waste of use value caused by the premature preventive maintenance, it is determined whether component i should be subject to opportunistic maintenance at time 3t, as shown in The reliability threshold when carrying out opportunistic maintenance for component 3t. In 3t. Assuming a total of ih opportunistic maintenance procedures conducted for component iuth  preventive maintenance of the multi-component system happens to be the jth preventive maintenance of component i, the downtime of component i is:Suppose that l as an example, the downtime of component l when the yth preventive maintenance of the multi-component system is expressed as Equation (18). The preventive maintenance cost of component l is expressed as Equation (19).At the same time, other components are assessed for whether it is necessary to perform preventive maintenance. Taking component y-th preventive maintenance procedure of the multi-component system is:Then, the downtime of the The total downtime of preventive maintenance of a multi-component system during the warranty period is:The total downtime of corrective maintenance of a multi-component system during the warranty period is:The warranty cost of a multi-component system during the warranty period is:Availability is an important index to measure the proportion of normal working time within the total time of multi-component systems throughout a certain period. Users have high requirements for the availability of multi-component systems. The availability of a multi-component system during the warranty period is:Under availability constraints, the opportunistic maintenance model of multi-component systems aiming for the lowest warranty cost during the warranty period Here, i, the greater the possibility of performing preventive maintenance on or the replacement of component i in advance. The original preventive maintenance plan of the component is changed, and the preventive maintenance interval will become longer, which increases the probability of unexpected failure of the component. Therefore, the corrective maintenance cost and corrective maintenance downtime loss of the system will become higher. The smaller the value of the reliability threshold i, the more the system will be subject to excess preventive maintenance, which will lead to higher preventive maintenance costs and increased downtime of the system. The reliability threshold The greater the value of reliability threshold According to the single-component preventive maintenance interval determined in Step 1: The utilization rate range of a multi-component system is divided into b intervals on average, the probability bP of each interval is calculated according to the distribution of the utilization rate, and the mean value g(r) is the probability density function that the utilization rate obeys:x is the lower limit of the utilization rate interval and y is the upper limit of the utilization rate interval.Step 2: On the premise that the utilization rate is in of a single component; then, the preventive maintenance interval Step 3: The preventive maintenance time a is the number of imperfect preventive maintenance procedures performed on the system, and 0t = 0, a = 1, respectively.Step 4: Preventive maintenance is carried out on component i at time at, and other components are assessed as to whether it is necessary to conduct opportunistic maintenance  is used to calculate the downtime Step 5: The array at time is deleted. If a +t1 is deleted, the time after at is marked as a +t1.Step 6: The corrective maintenance downtime of each individual component in the  is calculated, and the corrective maintenance downtime of all components is summed to obtain the corrective maintenance downtime a−t1, at).Step 7: The total downtime a−t1, at) is calculated. a = a + 1. Return to Step 4 until Step 8: The corrective maintenance downtime ot the end of the warranty period is calculated. Then, the total downtime Step 9: The expected warranty cost The expected warranty cost Step 10: The availability is judged as to whether it meets the constraints. If so, the cost value is stored as well as the corresponding reliability threshold when carrying out opportunistic maintenance; if not, the cost value is eliminated as well as the corresponding reliability threshold when performing opportunistic maintenance. The reliability threshold when conducting opportunistic maintenance is changed to find the scheme that minimizes the warranty cost.The algorithm flow chart is shown in An example of a power transmission device produced by an engine manufacturer in China illustrates the application procedure of the proposed model. All data are from the manufacturer’s historical data or reliability tests. Except for the parameters included in the failure rate function, other parameters can be obtained directly from the manufacturer. The power transmission device was composed of four key components: the valve train component, the lubrication system component, the fuel supply system component, and the starting system component. The component numbers are shown in The manufacturer is responsible for the power transmission device warranty. Generally, preventive maintenance is carried out independently for each component according to the failure law of each component. However, the manufacturer responded that this maintenance method may lead to wasted maintenance costs, mainly because the preventive maintenance work times of some components are similar. If this preventive maintenance is conducted at the same time, the total number of preventive maintenance procedures of the system will be greatly reduced, thus, reducing the maintenance cost. In view of this, the opportunistic maintenance strategy proposed in this paper is adopted in this case, and the advantages of opportunistic maintenance are demonstrated through comparative analysis. The logic of this case is shown in Each component has a trend in degradation in the two dimensions of calendar time and driving mileage. As the mileage increases, the failure rate of the power transmission device will increase; on the other hand, when the engine does not work for a long time, the phenomenon of dry friction or semi-dry friction is more serious when it is restarted, which will accelerate the wear and degradation of components. Therefore, the calendar time and driving mileage affect the reliability of the power transmission device at the same time. Preventive maintenance should pay attention to the calendar time and driving mileage at the same time. The distribution of the service life of each component is as follows:Z is 500, and the estimated parameters of the failure rate function of each component are shown in 4 KM / year) follows the uniform distribution of (0.1 ~ 10). Thus, explicit expressions of g(r) and G(r) are:The parameters of the above formula can be estimated by the method proposed in 4 KM. During the warranty period, the regular preventive maintenance strategy is adopted for each component of the power transmission device, and the corrective maintenance (minimum maintenance) strategy is adopted for unexpected failures within the preventive maintenance interval. When the preventive maintenance of each component reaches a certain number of times, the preventive replacement is implemented. The increasing age factor and decreasing failure rate factor of each component are the same, both At the same time, it was known that the power transmission device had a two-dimensional warranty period of 2 years, 2 × 10In this case, the failure dependence coefficient is estimated by the manufacturer’s experience. The failure dependence coefficient matrix of the power transmission device is:Other parameter settings are shown in It was known that the power transmission device had a two-dimensional warranty period. A year is calculated as 365 days. Firstly, according to the model detailed in r as 1 × 104 KM/year as an example: when opportunistic maintenance is not considered, the preventive maintenance intervals of each individual component for the dimensions of calendar time and mileage are shown in 4 KM/year, the preventive maintenance time of each component for the dimensions of calendar time and mileage  can be calculated as shown in As can be seen in A total of 21 preventive maintenance activities were carried out for the power transmission device. After calculation, the total expected warranty cost of the multi-component system was CNY 535,820 and the availability of the multi-component system was 0.8123. With a high number of preventive maintenance procedures, the preventive maintenance downtime increases, resulting in the waste of warranty cost. It can be seen from Firstly, the genetic algorithm proposed in Without considering the opportunistic maintenance, according to the maintenance plan of each individual component detailed in In order to demonstrate the advantages of GA, it was compared with the particle swarm optimization (PSO) algorithm and simulated annealing algorithm (SAA). Next, the PSO algorithm and SAA were used for optimization. For the PSO algorithm, the number of iterations was 270 and the population size was 100. The inertia weight of the PSO algorithm was 0.9, the self-adjustment weight was 1.49, and the social-adjustment weight was 1.49. The initial value of The minimum warranty cost calculated by the PSO algorithm was CNY 523,685, and the corresponding optimal Through comparison, it was found that the GA can achieve lower warranty costs, and converges earlier with higher operational efficiency; therefore, it has more advantages in solving this model.At this time, the maintenance plan of the power transmission device needs to be adjusted. According to the opportunistic maintenance strategy of multi-component systems detailed in Abbreviations, the maintenance plan of power transmission devices is adjusted, as shown in In Opportunistic maintenance combines preventive maintenance performed at different times, effectively reducing the number of preventive maintenance procedures of a multi-component system, thus, reducing the preventive maintenance downtime of multi-component systems during the warranty period and reducing the shutdown loss. Opportunistic maintenance not only reduces the number of preventive maintenance procedures, but also reduces the minimum number of maintenance procedures. This is mainly because opportunistic maintenance advances the time of preventive maintenance for components, which effectively avoids the occurrence of potential failures. The comparison of power transmission device warranty cost and availability between situations with opportunistic maintenance and without opportunistic maintenance is shown in As shown in When the failure dependence between the components of the power transmission device is not considered, i.e., when the failure dependence coefficient matrix It can be seen from 4 KM. The warranty period of one-dimensional warranties consists of one dimension, which is usually calendar time. When a one-dimensional warranty is adopted, the warranty area changes from a rectangle to an open area with an opening above, which means that the warranty period remains unchanged for 2 years, no matter how the utilization rate changes. The warranty period of two-dimensional warranties consists of two dimensions, one of which is calendar time and the other of which is usage. In this example, the warranty period of the power transmission device was 2 years: 2 years × 10As can be seen from TJ, the preventive maintenance time of each component is adjusted to an integral multiple of the preventive maintenance benchmark interval so as to achieve the result of combining the preventive maintenance of each component. Reference [This paper adopted the opportunistic maintenance strategy to combine the preventive maintenance of each individual component so as to reduce the warranty cost and improve the system availability. In addition to the opportunistic maintenance strategy, the grouping maintenance strategy is another widely used maintenance strategy. By introducing the preventive maintenance benchmark interval eference  adopted eference , the varIt can be seen from It can be seen from (1)The two-dimensional warranty cost of the power transmission device is significantly reduced, and the availability is significantly improved after implementing the opportunistic maintenance strategy, which fully verifies the effectiveness of this strategy.(2)The assumption of failure independence will lead to serious analysis errors and decision-making errors, meaning that it is difficult to provide support for the formulation of a warranty scheme.(3)The two-dimensional warranty method helps manufacturers save on warranty costs. With the increase in the utilization rate, the advantages of two-dimensional warranties are more obvious compared with one-dimensional warranties.(4)Compared with grouping maintenance, opportunistic maintenance has more advantages in reducing warranty costs and improving system availability.This paper considered an opportunistic maintenance strategy for two-dimensional warranty equipment based on the failure dependence of multiple components with the aim of minimizing the manufacturer’s warranty cost. Taking a power transmission device as an example, the results show that:In practice, in addition to imperfect preventive maintenance and replacement, the preventive maintenance strategy includes a function inspection strategy and a failure detection strategy. Function detection strategies assume that there is a process of functional degradation of components, and its theoretical basis is the delay time model; the failure detection strategy considers the existence of hidden failure modes of the equipment. Adding these preventive maintenance measures to the two-dimensional warranty service decision-making model will greatly improve the applicability of the model, but the modeling process will be more complex, and it is worthy of further study. In future studies, we will consider adding the concept of performance into two-dimensional warranty services. The core aim is to present the warranty service provider with certain rewards or punishments through evaluations of the warranty effect, so as to improve the enthusiasm of the warranty service provider and improve the warranty effect. Considering that it is highly necessary to purchase extended warranty services for some new complex equipment, it is necessary to study the extended warranties offered. Making decisions on complex two-dimensional warranties for equipment with failure dependence will be the focus of future studies.


With the increase in the number of waste power batteries and the occurrence of related environmental problems, battery recycling is receiving extensive attention. Driven by economic benefits, many companies have begun to deploy the waste battery processing market and government subsidies also play an essential role in battery recycling. Considering the vehicle company outsources processing tasks or invests in research and development (R&D), this paper studies the optimal decision-making problem of the supply chain under government subsidy to the battery manufacturer or the battery manufacturer. The research finds that: (1) For the government, when the vehicle company outsources processing tasks, compared with subsidizing the vehicle company, the total recycling volume when subsidizing the battery manufacturer is higher. When the vehicle company invests in R&D, the total recycling volume under different government subsidy strategies is equal. (2) The vehicle company’s decision is only related to its processing costs; when the unit processing cost is low, the vehicle company’s profit under the strategy of investing in R&D is higher. However, when the unit processing cost is high, the profit of outsourcing processing tasks is higher. (3) With increase in unit subsidy and decrease in unit processing cost, the total recycling volume will increase. These findings can provide decision-making help for the government in formulating subsidy policies and the vehicle company in determining processing strategies in the future. People’s increasing concern for the environment has recently promoted the rapid development of new energy vehicles (NEVs) around the world . As a keUnder the current situation of increasingly fierce competition in the forward supply chain, the reverse supply chain has become a new source of profit for enterprises. However, according to statistics, of the 74,000 tons of power batteries produced in 2018, only 5472 tons were recycled, with a recycling rate of only 7.4%, mainly due to the lack of recycling infrastructure and cost-effective processing technologies. In addition, effective regulations and policies are also of great significance in promoting the recycling of waste batteries and governments around the world are highly concerned about the management of waste battery recycling ,16. For Although research on recycling and processing waste batteries attracted people’s attention, most studies neglect the decision-making of the companies, i.e., whether companies will enter the waste battery processing market in the future. However, as the competition in the forward supply chain becomes increasingly fierce, more companies will face the issue of whether to outsource processing tasks or invest in R&D. Therefore, the decision-making problem for companies is worth studying. Considering that the battery manufacturers have battery production technology and that the difficulty of using raw materials and recycling and dismantling can be considered in the process of battery R&D, most previous studies regard battery manufacturers as the main body for processing waste batteries. In addition to the battery manufacturer, this paper also regards the vehicle company as the main body of processing waste batteries. The main reasons are as follows: investing in waste battery processing technology requires high cost and vehicle companies are generally in the core position in the reverse supply chain, thus their financial situation is relatively good. Besides, many vehicle enterprises, such as BMW, Volkswagen, BYD, etc., have begun to create a cascade utilization market of waste batteries. Therefore, based on the actual situation, vehicle enterprises can also be the main body for waste batteries processing. At the same time, government subsidies play a guiding role in recycling waste batteries. The selection of government subsidy objects also dramatically impacts the power battery closed-loop supply chain (CLSC). Therefore, in the actual implementation process, how government subsidy can optimize the operation of the CLSC is also worth discussing. Thus, considering the vehicle company outsourcing processing tasks or investing in R&D, this paper studies the optimal decision-making problem of the supply chain under the government or the company subsidizing the battery manufacturer, analyzes the impact of critical parameters on equilibrium decisions and compares equilibrium decisions. This paper intends to solve three problems: (1) From the government level, when the vehicle company determines the method of processing waste batteries, i.e., the vehicle company outsources processing tasks or invests in R&D, which subsidy strategy can the government implement to maximize the total recycling volume of waste batteries? (2) From the level of the vehicle company, when the government determines the subsidy object, i.e., the government subsidizes the battery manufacturer or the vehicle company, which processing decision can the vehicle company take to maximize profits? (3) What is the impact of changes in the unit subsidy and processing costs of the battery manufacturer and the vehicle company on equilibrium decisions?The contributions of this paper are summarized as follows: (1) Much existing research on waste battery recycling or processing has neglected the problem of waste battery processing decision-making by vehicle companies. This paper considers two situations of vehicle company self-processing and outsourcing processing tasks and studies the decision-making of the vehicle company; (2) Fewer studies have explored the impact of different government subsidy strategies on decision-making in the vehicle company. The findings of this paper can provide decision-making help for the government to formulate subsidy policies and for the vehicle company to determine strategies in the future.Research related to this study includes two main aspects: one is research on the decision-making when processing waste batteries by enterprises and the other is research on the impact of government subsidies on recycling waste products.Research on decision-making when processing waste batteries by enterprises includes two main aspects: research on enterprise’s investment in in R&D and research on enterprises’ outsourcing of processing tasks. Investing in R&D means that the enterprise is responsible for recycling and processing waste batteries at the same time. Ma et al.  designedOutsourcing processing tasks means that the enterprise is only responsible for recycling and outsources processing tasks to other enterprises. Some scholars regard cascade utilization enterprises, recycling and dismantling enterprises, and battery manufacturers as outsourcing enterprises: Zhang and Chen  studied Research on the impact of government subsidies on recycling waste products includes two main aspects: research on the impact of different government subsidy policies on recycling waste products and research on the impact of different government subsidy objects on recycling waste products.Regarding research on the impact of different government subsidy policies on recycling waste products, Zhang et al.  studied Regarding research on the impact of different government subsidy objects on recycling waste products, Ma et al.  studied The main research object of this paper is the supply chain composed of a battery manufacturer and a vehicle company, as shown in First, the following assumptions are made for the model:(1) In most cases, vehicle companies are “assembly plants” that do not have technologies for battery production, disassembly and processing, while battery manufacturers have battery production technology and can consider the use of raw materials and the difficulty of recycling and dismantling in the process of battery R&D. Therefore, battery manufacturers have more advantages in processing waste batteries . Conside(2) Referring to the function of the recycling volume of waste products by some scholars under mixed channels, it is assumed that the recycling volume of waste batteries by the battery manufacturer and the vehicle company is respectively :DBij=r0+The parameters involved in this paper are shown in Nomenclature.When the government subsidizes the battery manufacturer and the vehicle company outsources processing tasks to the battery manufacturer, based on the above assumptions the profits of the battery manufacturer and the vehicle company are expressed as:The equilibrium decisions are solved as:When the government subsidizes the battery manufacturer and the vehicle company invests in R&D, based on the above assumptions the profits of the battery manufacturer and the vehicle company are expressed as:The equilibrium decisions are solved as:When the government subsidizes the vehicle company and the vehicle company outsources processing tasks to the battery manufacturer, based on the above assumption, the profits of the battery manufacturer and the vehicle company are expressed as:The equilibrium decisions are solved as:When the government subsidizes the vehicle company and the vehicle company invests in R&D, based on the above assumptions the profits of the battery manufacturer and the vehicle company are expressed as:The equilibrium decisions are solved as:In the previous section, we built and solved four models that considered government subsidy strategies and the vehicle company’s processing decisions. Next, we discuss the research results from two aspects: sensitivity analysis and equilibrium decision analysis.In this subsection, we analyze the impact of the unit subsidy In this subsection, we conduct a comparative analysis of equilibrium decisions under four models. Firstly, under the fixed processing strategy, the total recycling volume of waste batteries and the profit of the vehicle company under different government subsidy objects are compared, as shown in It can be seen from It can be seen from In this section, an example will be used to simulate the impact of different government subsidy strategies on the total recycling volume of waste batteries From Firstly, the research on the decision-making of processing waste batteries by enterprises includes two main aspects: research on enterprise’s investing in R&D ,20 and oIn addition, this paper also puts forward relevant policy recommendations for the government and enterprises based on the above research results. The practical implications for the government are as follows: (1) When there is only a single company processing waste batteries in the supply chain, the government should subsidize this company; when there are multiple companies processing waste batteries, the government can subsidize any company responsible for processing waste batteries; (2) High processing costs are the main factor that hinders enterprises from processing waste batteries. Therefore, the government can consider subsidizing enterprises based on their processing costs so as to encourage enterprises to process waste batteries.The practical implications for enterprises are as follows: (1) Due to the high cost of processing waste batteries, vehicle companies with insufficient funds should not rashly enter the processing market. They can continue to deploy the recycling market and then outsource processing tasks to professional processing agencies; (2) For vehicle companies with relatively abundant funds, it is possible to consider establishing a waste battery processing branch by themselves. At the same time, it is also necessary to pay attention to improving the technology of processing waste batteries and reducing the unit processing cost; (3) For battery manufacturers, they can consider the recycling and processing of waste batteries at the beginning of power battery design, such as designing power batteries with higher safety, easier disassembly and lower content of toxic substances, which will be convenient for battery manufacturers in recycling waste batteries and will also reduce processing costs.This paper has some limitations. The model constructed is composed of a single battery manufacturer and a single vehicle company. In reality, third-party recyclers, secondary utilization companies and recycling companies are in the power battery supply chain. Therefore, further research should consider a supply chain that contains more subjects. In addition, to examine the battery recycling and processing market in real life, many companies have begun to explore ways to recycle and process waste batteries cooperatively to obtain higher profits. Therefore, the cooperation of supply chain entities can be further considered in further research. Finally, in addition to the subsidy policy, the government has also formulated other approaches to increase the recycling volume of waste batteries, such as deposit refunds, government rewards and punishment, etc. Therefore, in further research, the impact of different government policies on recycling waste batteries can also be considered.


Face recognition operating in visible domains exists in many aspects of our lives, while the remaining parts of the spectrum including near and thermal infrared are not sufficiently explored. Thermal–visible face recognition is a promising biometric modality that combines affordable technology and high imaging qualities in the visible domain with low-light capabilities of thermal infrared. In this work, we present the results of our study in the field of thermal–visible face verification using four different algorithm architectures tested using several publicly available databases. The study covers Siamese, Triplet, and Verification Through Identification methods in various configurations. As a result, we propose a triple triplet face verification method that combines three CNNs being used in each of the triplet branches. The triple triplet method outperforms other reference methods and achieves TAR @FAR 1% values up to 90.61%. Face recognition has become a popular technology exploited in many aspects of our life nowadays. A significant part of the research related to face recognition explores the visible spectrum of light, while other parts of the spectrum, including near and thermal infrared, are yet to be thoroughly investigated. The main cause of this fact may lay in the cost and accessibility of the equipment required to capture thermal face images of proper quality.Visible light imaging performs very well as long as the observed scene is properly illuminated, but this technology is not effective in low-light conditions. On the other hand, complementary thermal infrared imagery permits subjects to be observed even in pitch darkness. Combined thermal–visible face recognition may be a pivotal method to recognize subjects captured in low light conditions; however, the modality gap between thermal infrared and visible light domains needs to be filled in with the correct statistical algorithms trained using applicable databases.This paper reports on the study and implementation of thermal–visible face recognition for identity verification of subjects on-the-move. We propose a triple triplet method to compare face images in thermal and visible domains. This study is based on the outcome of an international project under the name of “Detecting document fraud and identity on-the-fly” (D4FLY), which received funding from the European Union’s Horizon 2020 research and innovation programme. The goal of the endeavor is to provide travelers moving around border crossing facilities with effective means of identity verification. The person verification scheme comprises two stages executed in order at the enrollment kiosk and in the biometric corridor. The enrollment kiosk, equipped with thermal and visible light cameras, is used to register new subjects in the system, while the verification is performed in a few-meter-long corridor furnished with a visible light camera only.Investigation of Siamese and Triplet architectures, together with VTI methods testing state-of-the-art algorithms for thermal–visible face verification;Examining spectral dependence of anchor images;Proposition of triple triplet method for using a specific set of CNNs for improved impostors and genuine subject distinction.This work’s primary contribution areas are as follows:In this section, we present works related to visible and cross-spectral face recognition between thermal infrared and visible face images, specifically for subject identity verification.Face recognition has gained superior popularity mainly in the visible spectrum due to easily accessible sensors integrated with popular electronic devices, including but not limited to smartphones and laptops. Current face recognition systems can outperform human perception capabilities in the visible spectrum of light, and are deemed the foremost tools available.Taigman et al.  proposedWen et. al.  proposedParkhi et al.  introducLiu et al.  proposedOn the contrary, thermal–visible face recognition is not as popular and widely studied as methods grounded in the visible domain, and results reported in this modality combination are lower in performance. One of the initial works in this field is focused on thermal–visible face identification . Hu et aChen et al.  presenteSarfraz et al.  presenteIn , KantarcFondje et al.  proposedNumerous works in this field have been performed using generative adversarial networks (GAN) to transform an image of one modality into an image in the second domain.Thermal–visible face recognition has also been addressed by employing GAN networks. In , Mallat Wang et al.  developeKezebou et al.  proposedImmidisetti et al.  proposedAnghelone et al.  proposedThe work of Cao et al.  presentsPoster et al.  have devFor this study, we developed a methodology that consists of data collection rules, algorithms selection, and experiment design. This multi-step methodology aims to develop several algorithms starting from dataset preparation and annotation, pre-processing, and face detection up to the face verification process, corresponding to the development of the feature extraction methods and decision functions.Since the extraction of features is proposed to be reused in other face verification architectures, several neural networks have been trained for identification purposes. These networks are the basis for feature extraction in various architectures considered during this study.(1)Database preparation and annotation. In this step, the datasets are combined and divided into training and testing splits. Another division used in this paper corresponds to the number of subjects and the presence of glasses. Since glasses are not transparent in the thermal infrared domain, they may impact the performance of thermal–visible face recognition. We decided to split the database according to the presence of glasses to assess this impact.(2)Development of face detectors. Wide study has been performed in this context to evaluate the possibility of using a single face detector for both spectrums. As a result, we developed two separate face detection algorithms for thermal and visible images, respectively.(3)Pre-training of CNNs with the ImageNet database;Training of all CNNs with a joint database of visible and thermal images;Training of all CNNs with separate databases of visible and thermal images.Training of CNNs for identification and feature extraction. The selected CNNs have been trained for classification purposes to learn the feature representation of thermal and visible images. The training procedure comprises the following steps:The following is a brief overview of the algorithm development phases:The trained CNNs will be further re-used for feature extraction in all the studied architectures.(4)Siamese-based methods;Triplet-based methods;Verification through identification approach.Development and testing of various methods including:(5)Analysis and selection of the best algorithm. This step corresponds to the calculation of performance metrics and the speed of processing.All the experiments have involved various settings of databases. Another goal was to evaluate how the data quality may impact the algorithm performance, especially for thermal infrared images.In order to validate the developed method in a controlled environment, it has been agreed that the thermal–visible face recognition system will be distributed across two components of the D4FLY system: an enrolment kiosk and a biometric corridor.(a)A thermal infrared sensor will be embedded into the enrolment kiosk;(b)The acquisition of face images on the move will be performed in the biometric corridor using a visible light camera.The proposed architecture was implemented in the D4FLY prototype and tested during multiple field test events.For the training, evaluation, and testing, several datasets were used during this study. The main requirement for all databases was to include respective images of faces acquired in the visible domain and thermal infrared. The datasets used for development came from our own repositories and from external sources. In this section, we briefly describe all the datasets that have been utilized during the study.The dataset consists of images of 31 subjects. Face images were acquired for each person at a distance between 1.5 and 4 m from the camera. After the subject had come towards the camera and stopped, the images were taken of the face in different head positions, but mainly in the frontal position . Images This dataset was collected at the premises of the Military University of Technology and contains visible and thermal infrared images of 40 subjects . VisibleDataset  is a pubThe Sejong dataset  consistsThis dataset  is a colThe datasets described in The testing dataset corresponds to the remaining 20% subjects of D4FLY and IOE_WAT supplemented with the first package of the Speaking Faces dataset and selected images of the Sejong Face dataset. Due to the large rotation of the face images in the Sejong Face dataset, only five frontal face images for each subject were used.The testing dataset was divided into two subsets composed of images presenting subjects with and without glasses. Finally, the number of subjects wearing glasses was 46, and not wearing glasses was 96. As part of the study concerns the use of the VTI method , a set oDuring this study, data augmentation techniques were applied for each of the datasets. Each dataset was composed of an equal number of images from thermal infrared and visible light domains.Since the study covers the face recognition task, it is important to balance the datasets to avoid or minimize the impact of various types of bias. The datasets used during this study were characterized in data bias terms. The statistical information on subject gender distribution is provided in As the datasets are not annotated with geographical information, we are unable to provide statistical information on the race distribution of the subjects.The study pursues a face recognition method that allows on-the-move object recognition with respect to the execution time. The GAN-based methods reported in the literature are computationally expensive and slow; therefore, we decided to apply the CNN-based approach. As a starting point, we prepared the face detection algorithm. Due to the modality gap between visible and thermal infrared images, we developed two separate face detection algorithms for each respective modality. As a result of our previous investigations, we used the Faster R-CNN with ResNet-101 to train two separate models. Study on the thermal face detection process is described in .We propose a modified triplet-based algorithm called triple triplet for thermal–visible face recognition. The standard triplet architecture requires three images to be processed simultaneously using analogous processing paths to compute the final scoring. In general, the three images are called positive, negative, and an anchor. The idea behind using an anchor image is to increase the separation of similar and dissimilar face images.Since the considered study uses two different modalities, the anchor image may be either visible or thermal infrared. The most popular trend in the literature indicates that the anchor should be of the same class as the positive image. Our in-depth study showed that the anchor image should be in the visible domain.D between them. Several existing distance functions were considered to calculate similarities between vector correlation distance, Spearman distance, Euclidean distance, and city distance (L1 distance). Moreover, we considered a triplet loss function, which can be calculated by the following formula:D(f(P),f(A)) is a distance function calculated between feature vectors of a positive image f(P) and an anchor image f(A), and D(f(N),f(A)) is the distance between the feature vectors of a negative image f(N) and an anchor image f(A).Each of the processing paths in the triplet architecture uses a similar CNN model, but in our case, each CNN is trained either on thermal or visible images, depending on the spectral range of the processed images. After the feature extraction stage, feature vectors are used to calculate the distance In this paper, we propose a triple triplet method by adding two additional convolutional neural networks into each branch of the triplet architecture. As a result of this modification, each branch uses three different CNNs to compute feature vectors simultaneously. These networks, namely, ShuffleNet, ResNet-18, and ResNet-50, aim to extract features from three face images at the same time, as presented in The proposed method generates nine feature vectors as a result of the feature extraction process. We have noticed that to achieve the highest performance, features should be extracted from the layer preceding the fully connected layer, which, in the case of the CNNs that we were employing, corresponds to the average pooling layer. TD1, TD2, the TD3) using the same formula as in (1). The triplet distances are calculated for each of the CNNs by the following formulas:As a next step, all feature vectors are used to calculate triplet distances , the number of neurons in the final Softmax layer equals two.In the final step, all the CNNs were trained using 80% of the subjects from the D4FLY and IOE_WAT datasets. All the datasets were gathered in the data stores.Testing experiments were divided into two parts, corresponding to subjects wearing and not wearing glasses. We used different datasets during each of the two experiments because of the availability of data presenting subjects wearing glasses. A composition of IOE_WAT, D4FLY, and Speaking Faces databases was used for testing the algorithms for subjects not wearing glasses, while the IOE_WAT, Speaking Faces, and Sejong Face databases were used for subjects with glasses.In this section, we present results divided into four parts corresponding to models and architectures developed and tested during the study. All the results are calculated using well-known biometric rates including the True Acceptance Rate (TAR) and False Acceptance Rate (FAR) of 1% and 0.1%, respectively. The overall dependence of TAR and FAR values is described by the ROC characteristics. During the tests, we calculated similarities between the feature vectors using various distance functions including cosine, Spearman, correlation, and Euclidean to compute the final face verification score.The first part of the study concerns using Siamese architecture. All the methods achieve better results for subjects wearing glasses, with the highest scores obtained for the joint dataset. The values of performance metrics achieved by the methods based on the Siamese architecture are far below expectations.In the second part of the study, we investigated the triplet-based methods in various configurations. We conducted two experiments corresponding to two cases. In the first experiment, we used thermal infrared images as anchors. The results of the experiment are presented in During the second experiment, we used visible images as anchors. The results of the second experiment are presented in The overall performance of the methods based on the triplet architecture is higher than the Siamese methods. The presented results indicate that the best results were obtained when the visible images were used as an anchor. Moreover, the TARs are mostly similar along different datasets with the best results achieved for the Speaking Faces database.Since the Verification Through Identification method performed very well in the thermal-to-thermal face verification task, the method was considered as a part of this study. The results obtained with the three best configurations of the VTI method are presented in As an outcome of the results that we achieved with the reference methods, we proposed a modification to the best-performing triplet-based algorithm, as presented in For the two best-performing methods, we prepared an additional analysis showing the impact of a head position on identity verification. The results are presented in Referring to The results presented in both tables show disproportion of the algorithms’ performance between male and female subjects. Moreover, the TAR value differences are also visible between the different testing databases. Both the gender and dataset bias are the result of relatively small and unbalanced datasets, which should be supplied with more subjects of both sex and uniform age distribution.The study aims to look for the thermal–visible face recognition methods that are able to process the images while the subject is walking through the biometric corridor. This condition is strictly connected to the processing speed. The average processing time of the proposed method is presented in In this section, we present the results of the method being trained without an essential part of the data. For this part of the study, we selected the two best-performing methods and trained them with single domain images. First, the methods were trained with a dataset containing visible face images. In the second step, the same methods were trained with thermal infrared images only. The hyperparameters of the training process are the same, as presented in In both cases, the methods were evaluated with a testing dataset composed of thermal infrared and visible images. The results of the ablation study are presented in Cross spectral face recognition using thermal and visible face images is not a well-explored field of research, mainly due to high-priced equipment and the relatively limited number of free image databases ready for use. Numerous manuscripts have reported the results found in the field of GAN methods, which can generate visible images based on thermal infrared imagery. However, these methods are not suitable for on-the-move verification due to their long processing time.In this work, we present the results of our study in the field of thermal–visible face verification using four different algorithm architectures with a proposed triple triplet method, which combines three CNNS being used in each of the triplet branches.In our study, we investigated several different algorithms in various configurations including the Siamese, Triplet, and VTI methods. For the training and testing purposes, we used several publicly available datasets composed of corresponding thermal and visible domain images.Since all the methods that we investigated used the same CNN models for feature extractions trained in the same conditions and configurations, the presented results are easily comparable. Our proposed triple triplet method achieves TAR @FAR 1% values up to 90.61%, depending on the testing dataset. Compared to the traditional triplet-based method, the triple triplet method increases the TAR value from about 65–67% to 73–77% when tested on the joint dataset. Compared to related works in the field, the proposed triple triplet outperforms the corresponding methods. Most of the works listed in the related works section are focused on the identification task, and thus are not directly comparable with our undertaking. The work of Chen et al. , the GANAll the algorithms were fourfold tested for performance, partitioned by an attribute related to glasses put on or removed, and another for frontal or rotated head positions. As anticipated, all the methods achieve better results for subjects not wearing glasses, since glasses are not transparent in the thermal infrared domain and significantly reduce the number of facial features when being worn. We also noticed a large variation in the results when testing with different datasets. Since the joint dataset was used during the training process, this fact may indicate a hardware bias, or a dataset bias in general, which is a result of a relatively small training dataset.The presented study shows that there is still a lack of publicly available large datasets of joint thermal infrared and visible face images. The quite high FAR values can be overcome when using large datasets during the training process. In addition, the study showed that glasses lower the method’s performance; thus, subjects should be asked to remove them before being checked.The presented study is considered as another step toward a fully functional thermal–visible face recognition method. In future works, we want to expand our dataset and propose new algorithms and methods.


Three-dimensional (3D) shape measurement for specular surfaces is becoming increasingly important in various applications. A novel orthogonal dual-frequency fringe is proposed in the specular surface shape measurement to overcome the phase jumping and discontinuities in spatial phase unwrapping. The fringe recalibrated high-accuracy phase information from its high-frequency fringe component with low-ambiguity phase information from its low-frequency fringe component. An improved Fourier transform deflectometry method based on the orthogonal dual-frequency fringe is proposed to measure 3D specular surface shapes. Simulation results showed that the orthogonal dual-frequency Fourier transform deflectometry (ODD) method could precisely reconstruct flat surfaces with an error of 2.16 nm rms, and concave surfaces with an error of 1.86  Specular surfaces such as reflective mirrors are widely utilized as crucial components in systems or applications in aerospace, precision instrumentation, the automotive industry, and the electronics industry ,2,3,4. RThe three-dimensional (3D) measurement and reconstruction of specular objects differ from diffuse reflecting objects, since the specific light reflected by specular surfaces can only be observed through specific angles, and surface shapes cannot be directly observed. Three-dimensional reconstruction algorithms for general scenes or diffuse reflecting objects do not perform well in terms of specular objects. Therefore, the 3D reconstruction and measurement of specular objects is also of great interest. Currently, 3D specular surface shape measurement methods are divided into two main categories according to the measurement principle: interferometry ,7 and deIn order to obtain the phase distribution, several fringe patterns in two orthogonal directions are usually required to complete the measurement process in the phase-measuring method ,19. CompTo improve the stability and the accuracy of phase unwrapping, a novel orthogonal dual-frequency fringe is proposed in the specular surface shape measurement to overcome the phase jumping and discontinuities in spatial phase unwrapping. The fringe recalibrated high-accuracy phase information from its high-frequency fringe component with low-ambiguity phase information from its low-frequency fringe component. To achieve a stable and efficient shape measurement of specular surfaces, an improved orthogonal dual-frequency Fourier transform deflectometry (ODD) method is presented in this paper. ODD can use the same system schematic with phase-measuring deflectometry and achieves the same accuracy while using only one image to extract phase information. An ODD fringe pattern simulation model is designed to simulate the fringe patterns deformed by different specular surfaces, and the performance of ODD is tested. An experiment based on a PMD measurement system was conducted to test the performance of the ODD system. The measurement experiment results also showed that the reconstructed shapes of both the flat mirror and the concave mirror measured by the ODD measurement system were highly comparable to those obtained by the phase-measuring deflectometry (PMD) method. This paper is organized in the following way: The deflectometry method applies vision techniques to detect the optical path and calculate surface slopes to reconstruct specular surface shapes. A basic measurement system comprises a hardware measurement platform and postprocessing algorithms. A general windowed discrete Fourier transform deflectometry measurement platform is demonstrated in A is the background intensity of fringe patterns, B is the magnitude intensity of sinusoidal components in the fringe pattern, Unlike most deflectometry methods, the ODD method focuses on the phase retrieval step and introduces the composite fringe pattern to improve the accuracy of phase information. The windowed discrete Fourier transform (WDFT) method ,28,29 isx- and y-directions. As is shown in The functions of the two frequency components are different. A typical orthogonal dual-frequency fringe pattern is shown in The image is transformed to the frequency domain via the two-dimensional (2D) discrete Fourier transform  to extraIn the frequency domain, there are two frequency peaks in both directions in the orthogonal dual-frequency fringe patterns, as shown in s filter , and g that the gray scale of the fringes varies for different screen pixels. It can be found from Equation (9) that the phase values of the same screen pixel under different frequencies can be obtained if one of them is known. Therefore, the phase map of the low-frequency fringe can be utilized to assist in the recalibration in the phase map of the high-frequency fringe. The first step of building the phase map of high-frequency fringe is to perform WDFT and IDFT to obtain the wrapped phase map and then apply the phase unwrapping algorithm to obtain the continuous phase map. A reference high-frequency phase map can be obtained from Equation (9) and is then used to check and correct the range of high-frequency fringes. This recalibrates the whole phase map and improves the accuracy of the WDFT deflectometry measurement. However, in practice, due to the constraints of the calibration conditions and the difference in measurement sensitivity between the two fringes, the calibration process may not run as the equation described; as the fast Fourier transform process is discrete, s is an arbitrary scale factor, and Please note that the radial and tangential distortion of the captured fringe was corrected in the image processing step according to the camera calibration method . Under ts origin , and theTherefore, the maps among every camera coordinate C:lculated . This prThe slope of the mirror point can then be obtained from the normal vector A simulation was conducted to verify the phase relationship between low- and high-frequency fringe components in orthogonal dual-frequency fringe patterns. In addition, a simulated deformed fringe pattern based on the real measurement system and the ideal shape of the specular surface was generated via a simulated pattern generation model . This frThe system configuration in this experiment is based on a typical PMD measurement system. It consists of an LCD screen with a resolution of 1920 × 1080 pixels, a high-speed camera with a resolution of 2048 × 2048 pixels at a frame rate of 30 frames per second (FPS) with a 16 mm lens, and a regulable flat stage. The fringe pattern displayed on the screen was generated by codes, and the camera was calibrated using Zhang’s camera calibration method . Before For the fringe pattern displayed on the screen, the composite fringe pattern can be expressed byIn order to examine the ideal performance of the ODD measurement system, based on the PMD measurement system simulation model , a fringTo compare the performances between the single-frequency fringe pattern and the orthogonal dual-frequency fringe pattern in the phase unwrapping process, we simulated a spherical surface shape as the tested plane and generated the deformed fringe via the fringe simulation model. Unwrapped phase maps extracted from high-frequency fringe components in both directions and corresponding recalibrated phase maps are shown in For the ideal flat simulated mirror, as shown in However, it is noted that the error of ideal simulated concave mirrors reconstruction was higher than that of the ideal simulated flat mirror shape. There are some possible reasons for this. Firstly, we noted that high error regions were distributed on the edge of the concave mirror, while the errors of the reconstructed flat mirror shape were homogeneous. This may have been caused by the height variation in the reflection characteristics of the concave mirrors. For the deflectometric method, when the height between the mirror shape and the reference plane increased, the difference increased. The edges of the concave mirrors were relatively high compared with the center, so there may have been high edges. Another possible reason is that the rotation matrix and the translation vector theoretically calculated from the camera calibration were applied to the shape measurement of the near-reference shape. When the height was large, the matrix and the vector may be not applicable to the edge of the shape, resulting in an increased difference. In addition, phase errors in the phase unwrapping step may cause the difference. The phase unwrapping accuracy in the concave mirror shape reconstruction is lower than that in the flat mirror shape reconstruction; even after the recalibration step, there were small phase jumps in the phase maps, which are shown in In order to test the performance of the ODD measurement system, we introduced a PMD measurement system as the hardware platform of the ODD measurement system. Note that the system configuration and parameters of the ODD measurement system were identical to the simulation. Both flat and concave mirrors were tested. As the only significant difference between PMD and ODD measurement system was the displayed fringe patterns, and PMD methods are well-known methods in specular surface shape measurement ,9,10,11,With similar measurement accuracy, ODD can achieve similar measurement accuracy to PMD while using only one sampled image, and the system components and architecture of the measurement system are identical. ODD provides a basis and opens new pathways for such methods to achieve dynamic surface shape measurement on specular surfaces.This study presented a new deflectometry method based on the orthogonal dual-frequency fringe pattern and the windowed Fourier transform method to reconstruct the 3D shape of specular surfaces. The linear relationship between the phase maps of the low- and high-frequency components was verified. The recalibration from the orthogonal dual-frequency fringe effectively reduced phase unwrapping ambiguities in high-frequency phase unwrapping. The simulation results showed that ODD could reconstruct the flat mirror shape at a nanometer scale, and the concave mirror shape reconstruction with ODD was at the micron scale. The experimental results showed that the ODD method achieved the same accuracy in measuring flat and concave specular surfaces compared with the phase-measuring deflectometry method. Both the simulation model and experimental results demonstrated the viability of the proposed method. However, it has certain limitations in terms of sources of errors in the simulation for the concave mirror shape measurement. Further work needs to be carried out to estimate the robustness of the ODD method, and continued efforts are needed to make the ODD method more accessible to the real-time shape measurement for specular surfaces.