The Internet of Medical Things (IoMTs) is gradually replacing the traditional healthcare system. However, little attention has been paid to their security requirements in the development of the IoMT devices and systems. One of the main reasons can be the difficulty of tuning conventional security solutions to the IoMT system. Machine Learning (ML) has been successfully employed in the attack detection and mitigation process. Advanced ML technique can also be a promising approach to address the existing and anticipated IoMT security and privacy issues. However, because of the existing challenges of IoMT system, it is imperative to know how these techniques can be effectively utilized to meet the security and privacy requirements without affecting the IoMT systems quality, services, and device’s lifespan.This article is devoted to perform a Systematic Literature Review (SLR) on the security and privacy issues of IoMT and their solutions by ML techniques. The recent research papers disseminated between 2010 and 2020 are selected from multiple databases and a standardized SLR method is conducted. A total of 153 papers were reviewed and a critical analysis was conducted on the selected papers. Furthermore, this review study attempts to highlight the limitation of the current methods and aims to find possible solutions to them. Thus, a detailed analysis was carried out on the selected papers through focusing on their methods, advantages, limitations, the utilized tools, and data.It was observed that ML techniques have been significantly deployed for device and network layer security. Most of the current studies improved traditional metrics while ignored performance complexity metrics in their evaluations. Their studies environments and utilized data barely represent IoMT system. Therefore, conventional ML techniques may fail if metrics such as resource complexity and power usage are not considered. The Internet of Thing (IoT) is a fast growing technology by which the infrastructures, computerized machines, physical things, applications and individuals are allowed to connect, communicate, capture and exchange information through networking . Hence, Medical devices and biosensors are responsible for capturing body’s vital signs, and transferring a huge real-time raw biological data such as heart rate, brain signal, temperature of the body, and glucose level in blood . These rThe data analytics are conventionally performed at the cloud. However, cloud computing faces the issues of delay and privacy. For this reason, the term of fog computing at gateways (fog devices) was introduced in 2017 . This apThe IoMT devices are vulnerable to traditional and zero-day attacks. This is mainly due to the lack of existing security protocols and measures in the fabrication of the devices in addition to the nature of the devices and IoT network. The devices are very tiny, where their computational resources and batteries cannot handle computation of cryptographic and existing heavy security measures. Furthermore, the network of the IoMT is heterogeneous, which is composed of different protocols at each layer, making one security solution not applicable to all devices. It was reported in an estimation made by the Statista  that the number of medical IoT devices in the European Union (EU) could reach 25.8 million devices by 2025 . In addition to a steady increase in the number of smart medical devices and benefits from the low price of wireless sensors, the security and privacy issues have become the main concern . BesidesConfidentiality: It limits unauthorized access to certain information and it guarantees the protection of confidential information. Unauthorized access may lead to data leakage and sometimes life-threatening situations . Other metrics such as privacy, and non-repudiation are also essential for the IoMT system . The folA threat that risks any of the aforementioned pillars of the IoMT security and privacy is considered an attack. Machine Learning (ML) is a branch of Artificial Intelligence (AI) which learns from data and experiences without being explicitly programed . ML can supervised, unsupervised, and semi-supervised algorithms. The Supervised technique deals with labeled data, in which classes or labels are known. There are two forms of supervised learning which are classification and regression  . Examples of these techniques are K-nearest Neighbor (KNN), Self-Organizing Map (SOM), and Latent Dirichlet Allocation (LDA)  has emerged recently, which is an advanced form of Neural Network. It has several layers of artificial neural network that mimic the working principle of human brain in processing data for detecting objects, recognizing speech, translating languages, and making decisions. DL can perform feature selection/extraction based on its learning process without the need of another method compared to traditional ML techniques that require an additional feature selection  . Since ion (LDA) . These m attacks . Based o attacks . Hence, ing data . Also, sing data . In addielection . DL comeelection . Auto-enelection . The useelection . On the election . Big datnd tools . A big dnd tools . It was nd tools . Very rend tools . ConcludAnomaly detection is the process of finding deviations from the normal data. Such anomalies are attributed to the occurrence of phenomena that do not obey a regular process. These irregularities are triggered by abnormal behaviors such as internal assault or false data injection attack , leadingAuthentication is the process of ensuring whether the IoMT user is correct, while authorization is associated with the extent of access given to each individual  . They arThe most common intrusion detection is network intrusion detection (NIDS) using network traffic analysis. If the log data of sensors and devices are used, it becomes host-based intrusion detection (HIDS) . Both tyIn the process of evaluating ML techniques in terms of accuracy, reliability, and applicability, some special metrics should be considered. The unique nature of the IoMT system with small size devices, heterogeneous network, and diverse protocols, has made the implementation of traditional security frameworks difficult for the medical companies. This in turn makes the IoMT system susceptible to different attacks. Recent advancement in the techniques and technologies of ML has led to achieve fruitful strategies to tackle the issues of the IoMT security. However, the IoMT system is facing more challenges than ordinary network systems.Internet of Medical Thing devices usually generate a large amount of streaming data . The divConsequently, understanding the current security and privacy issues of the IoMT system with their respective solutions using ML techniques is essential. Furthermore, it is significant to know the effectiveness of current deployed ML techniques and to understand their solution strength offered to the challenges of the IoMT system so far. We found that little attention has been paid in literature to elaborate on these issues. Therefore, in this work, a Systematic Literature Review (SLR) is presented, in which attempts are made to reveal the strong points and limitations of the works performed in this area followed by establishing robust improvement strategies. As such, three research questions can be generated as follows:RQ1: What is the current state of the art and direction of study in the IoMT security using ML?RQ2: What kind of tools and data are used for applying ML techniques in the IoMT security?RQ3: How ML techniques are effectively applied and what are their limitations?This study is intended for new researchers in the field, and for those who are keen to know about recent advances and limitations in the IoMT security issues followed by their solutions using ML approaches.In a review performed by In the current work, a systematic research design is generated. After creating a list of research questions, searching for relevant papers was started from different databases including IEEE, Web of Science, Springer Link, Scopus, Science Direct. Then, the most specific and relevant papers were extracted to answer the research questions. Later on, the selected papers were comprehensively screened and analyzed. Finally, the results were presented using different methods.A keyword-based search was applied by using different forms and combinations of Machine Learning, The Internet of Medical Things, Security and Privacy and their synonyms. Then, the synonym list was expanded during researching and a Research Information Template (RIT) was generated accordingly, as shown in In this work, two types of analysis were performed which include a bibliometric analysis and technical analysis, as discussed in the next subsections.To show the leading countries whose researchers working in the field of the IoMT and its security, each individual paper was examined according to the affiliation of authors. It was observed that the USA has made 30% of the papers among 21 affiliated countries, as shown in It is worth to mention that the majority of the papers were journal articles, contributing to 73% of the papers, while 21% of the papers were from conferences of the Institute of Electrical Electronics Engineering (IEEE) and only 6% of the papers belonged to book chapters as seen in The vast majority of ML related articles were about supervised ML, while the rest of papers were found to report on a combination of supervised and unsupervised ML, with few papers to focus on unsupervised ML. Deep learning was also used in some of the studies, while one study has used big data technology. The papers were further categorized based on the type of medical devices intended to get secured. Almost all studies have focused on the security of wearable devices, while few of them elaborated on the security measures for implantable devices. Only one study was found to focus on securing programmer devices. The targeted IoMT layers in most of the studies were device and sensor layers. Network layer was addressed in some studies, while cloud layer was reported by two paper. The papers were then classified into different subsections based on the approaches taken to tackle the security issues of the IoMT, as was discussed in the background section. In addition, an in-depth assessment was carried out through a critical analysis of the articles, demonstrating the strong characteristics and limitations of each study. In the following sections, we discuss on the findings of the aforementioned analysis.Herein, the review results of the current subsection related research articles are presented and analyzed, while the highlighted limitations of the studies are given in In a work performed by Despite the effective application of ML algorithms, they are generating high computational overhead on the low-power embedded frameworks. In another study , some MLIn another study made by In a pioneer work carried out by For the same purpose, a different approach has been proposed by Anomaly or malfunction sensor reading resulted from defective sensor nodes or produced by fraudulent foreign entities can contribute to medical error and even mortality in patients. Researchers  suggesteIt was recently  emphasizFurthermore, researchers  have impAdditionally, measured data of blood glucose sensor was examined in order to detect adversarial and accidental data modification intrusions . Here, OFurthermore, other research groups  presenteIn this section, results and analysis of the relevant papers on this topic are given, while the main features and limitations of the reported works are summarized in In a research made by In another work carried out by Furthermore, a study was performed by Hospital systems with huge number of patients require proper authentication method. For that purpose, Referring back to the individual authentication for daily use of wearable devices, researchers  used phoOn the other hand, Furthermore, in a study reported by Moreover, Moreover, Herein, the main achieved results on this topic that were reported in literature are given followed by a summarized comparison of the studies, as shown in Moreover, a multivariate correlation analysis was proposed for the IoMT attacks detection . TrianguAdditionally, in a study made by Furthermore, for the detection of anomaly based intrusion, a stacked autoencoder (SAE) was proposed by For the same purpose, researchers  used a mIn a recent work performed by Furthermore, Furthermore, with the aim of protecting the brain activity data from attacks in Brain–Computer Interface (BCI) systems such as EEG and fraud BCI program , which lets a malicious entity to obtain the user’s brain activities, researchers demonstrated that a ML method can anticipate relevant identity characteristics by studying resting-state EEG (rsEEG) records of a person’s brain activity . A complIn this section, the main outcomes of the reviewed studies are elaborated and discussed aiming at achieving fruitful answers to each of the research questions, respectively.One can see from Also, intrusion and malware attack detections received a considerable attention, counting for 35% of the reported studies . Nevertheless, the majority of the papers were found to have focused on NIDS. On the other hand, papers that used different strategies to solve the authentication and access control issues were found to take 28%. However, as authentication is computationally heavy for the IoMT devices, the current direction of research is to apply lightweight mechanisms  by usingMachine Learning methods are data dependent as they learn from these data overtime and decide intelligently based on their learning ability, amount and quality of the data. For this reason, in our study we have paid attention to this matter and we have analyzed all the selected research articles based on the type of data they used for decision making and learning process. It can be seen from To further answer the above research question, software and hardware tools used and reported in the reviewed studies are analyzed. Through a critical analysis of the current works, we can conclude that traditional ML techniques may fail if proper considerations are not given to some metrics such as computational complexity and energy usage. ML performance is reduced when few data are used. However, the IoMT devices are resource limited such as (IMDs) and using a huge data analysis on these devices is resulted in degrading their lifespan. For these reasons, one should take a balance between these two aspects. From our analysis of the reviewed papers, we have found that a small portion of the studies have used lightweight approaches. The pie chart shown in Based on the studies gaps that were assigned previously, it is found that majority of the studies have focused on improving the traditional evaluation metrics for their ML models such as high  and low . However, performance metrics such as memory (space), CPU, time, and energy overhead were neglected. Additionally, most of the techniques of attack detection were performed in an isolated offline environment. However, one should consider simulation/hardware implementation to represent a real IoT setting. Also, decentralized and hybrid approaches are much compatible for the IoMT than centralized one, which was again not common among the studies. Furthermore, we have noticed that the datasets used by the attack and intrusion detection methods were out of date and some of them do not represent the current IoMT system. Moreover, sensitive, and private data usage should be avoided in attack and anomaly detection tasks which was also given a minor attention. Therefore, there should be a trade-off among preserving privacy, high accuracy, and computational complexity. Another challenge that should be considered is the risk of adversarial attacks on ML techniques themselves. An attacker who has enough knowledge on how the ML techniques work with the data can manipulate the data at training or testing stage to fool the ML method. Hence, the proposed methods should be strong enough against the adversarial attacks.The underlying drawbacks of this research are: (i) Selection bias: This study focuses on literature research, which may have unintentionally omitted most recent non-scholarly advancements or scholarly articles that have not yet been published at the time of preparing this study. (ii) Publication bias: Some of the studies may have been discarded since their full paper was not accessible. (iii) Descriptive bias: Despite the efforts made to present the background comprehensively, this SLR is combining many topics that may need someone to refer to the external sources for a deep understanding.In this study, a comprehensive Systematic Literature Review (SLT) was given about the IoMT security and privacy issues and how Machine Learning (ML) methods are used for solving them. By examining the content of the study, including methods, good features, limitations, tools, and datasets, the designated research questions were answered. Findings of this study showed that ML techniques are effective in addressing the IoMT security issues with promising results. Majority of the studies was devoted to device layer or body area network security since attacks on devices such as IMDs are seriously affecting the patient’s health and life. The security solutions for such devices were sensor anomaly detection and device authentication and access control. Furthermore, securing the network layer was seen among the studies that used attack and malware detection strategies.Moreover, the tools and environment of the current works are a combination of network simulators and ML tools with more focus on the latter. Additionally, there is a lack of relevant datasets, especially in the intrusion detection. Most of the studies focused on improving the common ML algorithms evaluation metrics such as high accuracy and low FPR. However, since the IoMT devices are characterized with shortage in power and small memory and processor, there should be a balance between security and maintaining resources lifespan during the adaption of these solutions. We have concluded that traditional ML techniques may fail if proper consideration is not given to some metrics such as resource complexity, time complexity, and energy usage. It was noticed that a vast majority of the studies ignored these criteria in the evaluation of their proposed models. Therefore, ML techniques are vital in the application of the IoMT security. However, future studies should focus on how to use ML in a proper way to concede the nature of the IoMT.


One of the big problems of today’s manufacturing companies is the risks of the assembly line unexpected cessation. Although planned and well-performed maintenance will significantly reduce many of these risks, there are still anomalies that cannot be resolved within standard maintenance approaches. In our paper, we aim to solve the problem of accidental carrier bearings damage on an assembly conveyor. Sometimes the bearing of one of the carrier wheels is seized, causing the conveyor, and of course the whole assembly process, to halt. Applying standard approaches in this case does not bring any visible improvement. Therefore, it is necessary to propose and implement a unique approach that incorporates Industrial Internet of Things (IIoT) devices, neural networks, and sound analysis, for the purpose of predicting anomalies. This proposal uses the mentioned approaches in such a way that the gradual integration eliminates the disadvantages of individual approaches while highlighting and preserving the benefits of our solution. As a result, we have created and deployed a smart system that is able to detect and predict arising anomalies and achieve significant reduction in unexpected production cessation. Dynamic developments in informatics and intelligent technologies have led to the fourth industrial revolution. The ideas of this concept have been promoted in industry, but the leading innovators include the automotive industry and companies that primarily exploit the potential of automation and robotics.The basic components of the Industry 4.0 concept include, among others, Industrial Internet of Things (IIoT), wireless and intelligent sensors, big data, and artificial intelligence ,2,3.The development of advanced sensors allow the collection of huge amounts of data in real time, their processing and storing, followed by the analysis and discovering of new knowledge that can be used to predict the future behavior of industrial systems, including the identification and prediction of various anomalies.This principle is also widely used in the process of production equipment maintenance and has led to the acceleration of a special concept called Maintenance 4.0 . AccordiOne of the main tasks here is data which is actually information coming from the production process. Most of the equipment in current production lines are able to produce important information that can later be analyzed and the gained knowledge used more effectively. Using this knowledge, the standard preventive maintenance changes to predictive. This approach, often called Maintenance 4.0, can then more effectively address issues that arise, including those that are unknown in advance.The basic elements of Maintenance 4.0 are defined in ,8. From According to , one of For this reason, it is not only possible to use standard tools and techniques, but it is necessary to focus on the development of unique approaches that can solve the problem ,14,15. IAssembly conveyor is a single point of failure that can cause assembly process cessation,Bearing failure can occur any time on the assembly line despite the standard preventive maintenance,Decrease the production downtime using the predictive maintenance approaches.Our contribution is focused on solving issues that are associated with the production process in the automotive industry. Specifically, it is a conveyor used in the assembly process. The key issues that out paper addresses are:The assembly process is one of the most important processes in car production. Any problems that arise have a much greater impact on the flow of production than other processes. Any cessation of production, due to a problem in the assembly process, is quantified at such amounts of money that it does not leave any production manager calm.The conveyor is the backbone of the assembly process. Its trouble-free operation significantly affects the continuity of production. From time to time however, the bearing of one of the carrier wheels seize and thus the conveyor and, of course, production is stopped. The identification of a given stuck bearing in the wheel and its subsequent replacement often causes a couple of hour’s production outage.Despite well-planned and implemented preventive maintenance, this anomaly, this problem, occurs at random intervals and causes a problem that must be solved differently than the standard and known approach.The proposed platform intends to provide an early notification of possible carriers bearing failure, which will allow one to perform preventive maintenance. This prevents assembly line cessation, avoids unwanted downtime in production, and associated financial losses. One of the primary directions of the literature review is relevant for this study. It includes predictive maintenance as a special part of Maintenance 4.0, and also involves approaches related to anomaly detection.According to ,17, MainIn this document , the autIn these articles, the authors focused on the proposal of progressive approaches, which used new methods and technologies, such as the use of cyber-physical systems, artificial intelligence, or supervisory control systems in the architecture design ensuring more efficient implementation of the maintenance process.The production processes generate a large amount of data. In order to obtain an overview of its behavior, it is necessary to collect, store, and then analyze available data. IoT devices are suitable for collecting process data, as described in . Data coIn addition to standard behavior, anomalies sometimes occur during manufacturing processes that cannot be clearly identified with standard approaches. To identify them, it is necessary to use methods and techniques of machine learning. Based on the previous cited articles and on the practical experience, it can be stated that the most frequently used approaches include mainly neural networks.Predictive maintenance according to  can be dArticle  presentsThe survey  discusseIn this paper , the autAt present, companies emphasize the continuity of production and reduction of downtime caused by equipment failures. This can be achieved by applying the approaches offered by the Industry 4.0 concept. These include predictive maintenance. The trend of the cited articles clearly leads to the need for approaches that use intelligent devices, and are able to record and monitor the status of production equipment. It is possible to perform analyzes on the obtained data by methods of artificial intelligence and these will enable the identification of the possible non-standard behavior of the devices, which are subsequently eliminated by performing their maintenance.Our proposal focuses on the predictive maintenance of conveyors on the assembly line in the automotive industry. The assembly conveyer is used to transport car bodies during the entire assembly process. It consists of 59 carriers that move along a closed track with a length of approximately 1250 m. This track is leaded on two height levels, with four carrier’s lifts used to move carriers between them. The track contains places that can be used to temporary decommission carriers for maintenance or due to a failure on the assembly line.The carrier consists of a transport platform which is used for transporting prefabricated car bodies along a defined railway track. The individual carriers are controlled by their own Programmable Logic Controller (PLC), which cooperates with the company’s Manufacturing Execution System (MES) deployed in the assembly hall. This system provides, among other functions, data on the positions of the individual carriers. The carrier position, however, is not obtained directly from the carrier itself, but is identified only by the scanning gates.One of the main problems that the company faces during the assembly process is the cessation of the carrier (as a part of conveyor), which can ultimately lead to the cessation of the entire assembly process. For this reason, our proposed predictive maintenance approach aims to reduce production outages caused by problems with carriers.The most common problems of carrier cessations are caused by the jamming or seizing of a wheel bearing, which carries the entire weight of the transport platform including the car body and workers who perform the assembly process. It should be noted that the transported car body already contains most of the car’s components, such as engine, gearbox, steering subsystem, interior equipment, and so on. The platform is further stabilized by a guide bearing, which keeps it on a railway track, serving to lead it along a specified path in the assembly hall.The main carrier wheels are connected to the supporting frame of the transport platform using the bearing. The most common problem that occurs is a wheel bearing failure caused by the jammed bearing, due to wheel misalignment, dirt contamination, insufficient lubrication, etc. The consequence of this failure is an increasing bearing temperature to the level of its destruction, which is verified by people from maintenance. Therefore, the temperature increasing can be used as pre-fail parameter to predict the wheel bearing failure. This failure is also accompanied by a whistling sound, which can be heard from a certain frequency in the assembly hall.The company MES system contains only approximate information about the location of individual carriers, the task of maintenance crew is therefore to visually identify the damaged carrier. After its correct identification, it is necessary to tow the carrier to the nearest service point and replace the damaged bearing, usually together with the wheel. Time required to tow the carrier to the location is given by its distance to the nearest available service location and the time required for the replacement itself. The total time required to eliminate the fault varies from 30 min to 150 min. The consequence of the bearing failure is the cessation of the assembly conveyor behind a given damaged carrier which can lead (and from time to time leads) to a complete cessation of the assembly process.Other data that could be used for predictive maintenance are not found in the company MES system, given that the PLC focuses only on the control of the carrier’s movement along a predefined path.For using in predictive maintenance, we decided to measure the temperature of an individual carrier wheel bearings, which should be a sufficient parameter to detect well in advance an emerging problem in the monitored bearing. Providing timely information about a potential bearing problem will give the maintenance crew additional time to inspect the carrier itself.If the maintenance crew evaluates that the identified problem can cause a wheel bearing jam, they will have enough time to prepare for the replacement and to park the carrier in a suitable position at the service location where the replacement will take place. The bearing and carrier wheel replacement, carried out in this way, can usually take place during the planned break of the assembly line workers, which does not cause the cessation of the entire assembly process.One of the important prerequisites for the implementation of predictive maintenance was the availability of data from the assembly conveyor line, necessary for the detection of anomalies and evaluation of error conditions.The collection of temperature data from the assembly carrier, was performed following multiple approaches. To evaluate usability of the Industrial Internet of Things (IIoT) technologies in the selected automotive environment, it was decided to build IIoT infrastructure for the data collection that will future proof the modern production environment . This daThe design and implementation of IIoT network has focused on the reliability of the network in the assembly hall environment. Therefore, we decided to implement a private IIoT network based on LoRaWAN (Long Range Wide Area Network) technology. This technology was selected after being based on the plant’s recommendations for adopting wireless network technologies.To achieve sufficient signal coverage on the whole assembly conveyor path, we installed two Kerlink Wirnet Stations  operatinIt should be noted that although the area we are covering is a single assembly hall, and the maximum ranges are on the lower end of all used device specifications, two IIoT gateway devices were required to obtain optimal coverage of the whole assembly line and to minimize data loss. This is mainly due to the underground passages that the assembly conveyor is using during travel. Using only one gateway, we detected periodical data losses from the IIoT devices. The insufficient coverage on various spots using only one device was confirmed by the Adeunis ARF8123AA —a field-For the temperature measurements on the assembly carriers, we utilized LoRaWAN IoT temperature sensors Adeunis ARF8180BA , which pAlthough the wheel temperature is a relatively time stable process, the most detailed sampling period of 100 frames per day was selected for the proof-of-concept phase. Using the calculation mode of the device, which sends an average value from the measurements performed over a given period of time, one year of continuous monitoring per battery were expected. Using this configuration, we obtained data from the measurements every 14.4 min.The specifications of the used temperature sensors, shown in Data collection from the IIoT network was further supplemented by stationary temperature measurements, on identified points on the assembly conveyor path. As the measurement points were identified, two assembly carrier lifts transfer the carriers from the lower to upper level and carrier after continuous movement. In the identified points is a high probability of carrier wheel temperature increasing during a bearing failure, since the carrier is in continues movement and the bearing did not cool down.One of these measurement points was selected in the proof-of-concept stage to perform the stationary wheel temperature measurements as well as the motor lift parameters, such as lift platform inclination, lift motors current consumption, temperatures, and vibrations, which were also used for predictive maintenance purpose. The proposed measurement infrastructure is shown in For the data collection from the sensors, we used Viltrus Electronics Data Logger RAY-3 in configuration RAY-3.124.1.0.111333.2.4.10.1 . The selThe 6 analog inputs were used to connect temperature, vibration, and tilt sensors. For the temperature measurements performed on the 6 carrier wheels, data logger devices marked “Ray101” and “Ray102” were used. For the data collection in our implemented application, we utilized the Modbus protocol for direct access to the measured values. After the proper data logger register configuration, this approach worked without major issues.For the contactless temperature measurements, we wanted to deploy infrared pyrometer sensors from Optris, namely OPTCSTCLT15 . All thrSince we were not able to fix these issues with the help of Optris support and decided to switch to the Calex PC21MT-0  non-contThese sensors were connected using the analog outputs to the data logger devices marked as “Ray101” and “Ray102”. Due to the custom nature of the carrier lift measuring position, custom sensor mounts were created to hold them in place. The measuring distance was around 170 mm for all 6 installed sensors.The most challenging step in using these sensors, was the proper placement and stability of the sensor mounts to ensure the repeatability of measurements. After the initial installation, the repeatedly measured values were in the ±3 °C range. After the sensor mounts modification, and proper calibration, we achieved an accuracy of ±1.2 °C at the ambient temperature of 26 °C.For carrier identification, Radio-Frequency Identification (RFID) technology was used. The RFID technology was preferred instead of a bar code scanner, due to the poor lightning conditions and continuous setting of impurities on the carrier sides. A RFID reader device was used mid-range SICK RFID RFH630-1000001 , its parThe RFID device was installed on a custom mount, within the 150 mm distance from the carrier side, to achieve error-free tag reading. For the device communication, we used the RS-232 serial interface.Various types of RFID tags shapes and mount types were tested, however the square shape with the metal base, mounted using industrial glue and secured using translucent adhesive tape, proved to be sufficiently durable.For the data collecting and processing on the site, we utilized the Industrial PC (IPC) AAEON NANO-002N-6100-01 , whose pThe IPC was supplemented by the Solid State Drive and 60 W power supply from the vendor and was mounted on the measurement site supplemented by the external fan for providing additional air flow. Although the IPC contains two network adapters, only one was used due to the connectivity available on site. The IPC was running the Ubuntu Server 18.04 LTS.For sound measurements, we selected the calibrated measurement micro-phone miniDSP UMIK-1 , its speFor empowering the bearing failure detection in our proposal, we decided to add sound analysis in to the implemented approaches. The monitoring, detection, and diagnostic of bearing faults from sound recordings could be achieved using different methods. Some of them are simple for use while others require sophisticated signal processing.Overview of methods of the bearings condition monitoring are well summarized in the basic works ,47,48. PThey may be generally based on measurements of vibration, acoustic analysis, temperature measurements, and wear analysis. Shocks and bearing faults can be analyzed either in the time domain or in the frequency domain .The time-domain feature extraction method is the most dominant one for rolling element bearings. Scalar descriptors, such as Root Mean Square (RMS), peak amplitude of vibration level, Crest factor, Kurtosis, Skewness ,53,54, dThe most widely-used frequency domain methods are spectral analysis around bearing defect frequencies ,59, freqBall Pass Frequency Outer (BPFO),Ball Pass Frequency Inner (BPFI),Ball Spin Frequency (BSF),Fundamental Train Frequency (FTF).Acoustics methods are also established as promising for condition monitoring and bearings fault detection ,67. ManyThese frequencies, manifested in both vibration and acoustic signals, are detectable with the appropriate processing techniques. The signal of early fault is often too low due to the fact that small localized damage can generate small periodic pulses. Over and above, the measured vibrational signals in the initial stage are disturbed by loud sounds.Wheel bearing failure on the roller element makes up a great part of the failures identified on the assembly conveyor carriers. The failure usually results in the wheel bearing seizure, blocking the movement of the carrier and resulting in the temporary shutdown of the conveyor. Therefore, the detection of the wheel bearing failure through any monitoring process is especially important.Online condition monitoring, proposed in our paper, is therefore highly desirable due to the potential issues that can be detected and rectified during scheduled maintenance. The proposed and implemented solution incorporates requirements for the design and implementation of the monitoring system, which will provide early warning of potential failures of the carrier wheel bearings, in order to minimize the occurrences of unexpected failures.The proposed system covers multiple processes, involving data collection and their processing, data storage, and anomaly detection process, as shown in the process diagram shown in The diagram outlines the processes required for the monitoring and anomaly detection of the wheel bearing failure proposed in our paper. Data collection using the IIoT devices is performed by the IIoT devices alone as is further described in Trigger-based bearing failure detection,Anomaly detection using a neural network,Bearing condition detection using sound measurements.The anomaly detection approach consists of following methods that were proposed and implemented in our solution:These methods can operate separately and are combined in our solution to provide a comprehensive overview of the bearing wheel status, detected anomalies, and bearing failures themselves. Used methods are described in detail in All through the implemented monitoring and detection system is fully automated, the results evaluation is in the hands of the maintenance crew, since the maintenance and repair scheduling as well as the detected anomalies evaluation is carried out by them.For all the installed sensors and devices, it was necessary to design a data collection and integration process to obtain suitable data for further analysis. Since we utilized two data collection technologies, this approach can be divided into two separate designs that both integrate their data into the common data storage. As the main data storage, we utilized Yandex ClickHouse column-oriented database that can be used for efficiently storing and accessing time-series data.For the collection of data originating from the LoRaWAN devices, we designed and implemented the architecture shown in The data measured on the IIoT devices are transferred through the LoRaWAN network to the available Kerlink Wirnet Station in range. Since we are using multiple stations with overlapping coverage, receptions of the same data on both stations are quite common. Deduplication of received data was therefore required and performed in data collection application implemented in Python.To achieve near real time data storing, we utilized the Message Queuing Telemetry Transport (MQTT) protocol that is provided by the Kerlink Wirnet Station. The MQTT broker was utilized in the middle for data buffering and future expansion possibilities. In our proof of concept, we utilized the lightweight Mosquitto MQTT broker that met all requirements.For the transfer of MQTT data into the database server, a custom Python application using paho-mqtt library was created. This application was subscribed into the MQTT broker topics and transformed the obtained message payloads into the measured values.Sample data from the Adeunis devices, received by the Kerlink gateway, are shown in To decode the Payload value, official DataFrame documentation of the Adeunis devices was used. The device payload structure, shown in This conversion was supplemented by decoding the Status bits, to identify error states and low battery warnings, since the identification LEDs on the devices are not accessible during use and the devices do not provide any other status information.For the data archiving and debugging purposes, all the data received from both LoRaWAN gateways are stored on the File Transfer Protocol (FTP) server, using the options provided by the Kerlink Wirnet Stations. This process is performed periodically every 60 min. The target FTP server was installed on a CentOS server in the plant’s infrastructure.The data obtained from the stationary measurements shared the data collection infrastructure with the carrier lift sensors for the lift platform inclination, lift motors current consumption, temperatures, and vibrations. However, in our paper, we solely focus on the carrier wheels temperature measurement.Since we decided to incorporate the sound recording for the carrier bearing failure identification, we installed the measurement equipment near the stationary temperature measurement. The data collection and integration architecture for the carrier wheels temperature measurement and carrier sound recording is shown in Due to the shared infrastructure, temperatures of the front and rear carrier wheels are collected by the Ray Data Logger marked as Ray101 and middle carrier wheels are collected by the device marked as Ray102. The respective analog inputs on the Ray Data Logger device were properly configured for the input types and ranges of the used temperature sensors.For the individual analog input values, accessed through the Modbus protocol, was also required to properly set the Ray Data Logger registers. It should be noted that although the Ray Data Logger can perform basic input value transformation, the obtained Modbus values are always in raw input format.out is the measured current on sensor output, MaxValue is the maximum measurement range of the sensor, 20 mA is the largest output current of the sensor, and 4 mA is the lowest output current of the sensor.For the transformation of the measured current value into the measured temperature for the Calex sensor, we used:The RFID reader device is connected to the Industrial PC using the standard RS-232 connection. The measurement microphone is connected directly into the USB port on the IPC.The IPC runs the data integration application, implemented in Python programming language, for collecting and integrating the measured data. For the data reading in designed application, we used the PyModbus library for Modbus communication, PySerial library for serial communications, and PyAudio library for sound recording.For data backup and debugging purposes, we utilized the data logger function to store the data every 30 min on the FTP server. This server was shared with the IIoT gateway backups.For the stationary measurement of the carrier wheels temperature on the carrier lift, a custom measurement algorithm was designed, as shown in The measurement algorithm is designed around three stages that are looped indefinitely. The first state focuses on the RFID tag reading, since the valid tag is the signal for starting the carrier wheels temperature measurement. The RFID reader uses a serial bus with a 9600 baud rate. Since the RFID reader uses a serial bus for communication, the RFID tags are validated against the tag pattern and also against the RFID tag list that contains all installed RFID tags paired to the carrier ID number. When the tag is read and validated, the program is switched to the second state.In the second state, the program reads the data from the Ray Data Logger registers and converts them into the measured values, based on the type of sensor. For the used temperature sensors, the value is converted using Equation (1). The temperature is measured every second for five times, to increase the accuracy of the measurements. When the measurements are completed, the measured values are used to calculate minimal, maximal, and average values and are subsequently stored in the database. It should be noted that the saving of data into the database is performed asynchronously, to minimize the unpredictable events influence on the application logic. Finally, the algorithm is switched to the third state.In the third state, the measurement algorithm waits until the carrier starts to move away from the lift. The RFID tag reading must fail three times before we declare the carrier as moved. This is due to the tag reading issues despite the fact that the carrier RFID tag was still present. When the carrier has moved, we start the sound recording that is performed asynchronously in the separate application thread.The audio sample recorded for each carrier has 58 s and is recorded as a mono channel with a 16-bit word size and sampling rate of 48 kHz. For the recording in Python, we used the PyAudio library that provides the wrapper over the standard Linux ALSA library. One of the biggest issues of the PyAudio library was the underlying ALSA library written in C programming language. During continuous deployment, multiple issues arose caused by the C errors not captured and handled properly by the PyAudio library. Therefore, we had to implement a custom ALSA error handler, shown in It should be noted that we had omitted the argument parsing using the libc Python library and regular expression error matching in the code sample, for better clearness and simplicity. This code is usually recommended for ALSA device initialization, in our proof of concept we found that it is also necessary for audio stream recording. This was due to the occurrences of random ALSA errors during the recording. These errors occurred randomly, without any configuration or library version changes in the system.The recorded audio samples were stored on the FTP server, shared with the Kerlink, and RAY-3 data backups. One recorded sample has an average size of 5 MB and we kept all recorded samples stored for further audio analyses. To save the space, audio files older than one month were automatically deleted using scheduled Bash script.Due to the fact that the IIoT temperature measurements were implemented as first in our proof of concept, we could use this data for further analyzes. From this data we removed bearing failure records and assembly downtime during the weekends, to represented the carrier wheel temperature values measured during normal operating conditions.For the trigger-based bearing failure detection, using the carrier wheel temperature measurements, we decided to use common approach by using the wheel temperature threshold. The carrier wheel temperature was measured using the external contact temperature sensor on the Adeunis IIoT device.From the IIoT data obtained during a period of two weeks collected in February 2020, we created a histogram, shown in To compensate the data errors in the early stage of the proof of concept, we collected minimum, maximum, and average wheel bearing temperature values. In the proposal we decided to use the averages of measured carrier wheel temperatures, provided by the IIoT and stationary devices.Fixed temperature difference;Difference against the average wheel temperatures of the carrier;Difference against the wheel temperatures on previous carriers;Difference against ambient temperature.For detecting the carrier wheel temperature anomalies for predicting the bearing failure, we considered multiple approaches. These include for example:We evaluated proposed trigger-based detection approaches and performed tests using the collected data to select the most suitable for detecting wheel temperature anomalies. Although the fixed temperature threshold detection method worked, it required manually changing the detection value due to the unstable temperature in the assembly hall. Evaluating the difference against the wheel temperature on other carriers seemed promising, however this approach generated a large number of false positive or false negative detections before and after daily breaks in production. This was caused by the stationary carriers, i.e., the carrier wheels cooled down during the daily break.Therefore, we selected the difference against the average temperature of other carrier wheels and difference against the ambient temperature for trigger-based detection implementation. The detection parameters were specified based on the statistical analyses of collected data. These parameters were subsequently tweaked using following months data.The selected approach was also further supplemented by the data from the stationary temperature measurements. The correlation matrix in To implement the trigger-based carrier wheel temperature anomaly detection for predictive maintenance, we proposed an algorithm for evaluating data stored in the database and notifying assembly hall maintenance. The algorithm was designed to provide possibilities for detecting and evaluating any of the proposed methods. This was done by using the lambda expression for trigger function definition, as shown in the detection configuration code in The query parameter specifies the query used for obtaining the data from the database for the trigger function. The detect dictionary structure specifies the table columns used for the detection, and the lambda function that is evaluated. If the lambda function is evaluated as true, the detection was successful. The headers item defines array of database columns that will be added to the detection record for further processing.This algorithm was subsequently implemented in the application, developed in Python programming language and performed scheduled data evaluation every 15 min. The detected anomalies in the data were labeled and further processed by email and SMS notification functions to notify assembly hall maintenance.The results of the implemented trigger-based detection approach can be clearly seen in The bearing failure on wheel TR1, in this example, was detected at 19:34  value and the normal operation was represented by OK  value.Since there are storing minimum, maximum, and average temperature values from each carrier wheel measurement we had to evaluate which of these parameters group was more suitable for neural network inputs. The dependencies in the data set were analyzed using importance plot analysis, shown in Based on this result, we decided to use the minimal and average temperature values for each carrier wheel as input parameters for our neural network. These parameters were supplemented by the ambient temperature measured in the assembly hall, since the wheel temperature is directly dependent on the ambient temperature, as shown in It should be noted that the number of bearing failures that occurred during data collection was quite small in relation to the total amount of data records. Due this fact we decided to create data subset using Stratified sampling, which would mitigate the impact of data imbalances on bearing failure prediction ,73,74.The input data set was divided into training, testing, and validating sets in the ratio of 60%—20%—20%. Using the Automated network search function, provided by the TIBCO Statistica data science platform, several neural networks for the predicting of the bearing failure were designed. The obtained MultiLayer Perceptron (MLP) neural networks differed in various numbers of neurons in the hidden layer, type of activation and error functions, used training algorithm etc. Based on achieved performance in all subsets and their balanced variance, the most suitable neural networks are shown in For the suitable neural networks, we created confusion matrixes and calculated selected performance evaluations parameters. To empower the validation data set we had to decide to supplement it with data from the following five weeks during which 16 bearing failures occurred. This resulted into 87 carrier wheel temperature measurements to be labeled as Failure.1 ScoreF representing harmonic mean of the recall rate and precision as:True positives, False positives, True negatives, and False negatives are calculated from the neural network prediction spreadsheet.From the performance evaluation parameters, we calculated the neural network precision, recall, accuracy, and After the parameter evaluation, the neural network labeled MLP 13-18-2 with 13 neurons in the input layer, 18 neurons in the hidden layer, and 2 neurons in the output layer, was selected as most suitable. The confusion matrix of this neural network is shown in The selected performance evaluation parameters of the most suitable neural network are shown in The selected neural network is able to predict the bearings failure of the carrier based on the temperature of wheel bearings and the temperature in the assembly hall. During several tests with different input data sets, a neural network was selected capable of predicting the failure state of the carrier wheel bearing on average 4 to 6 h before the probable failures. Based on these evaluations, the selected neural network was accepted as a suitable to integrate it into the designed application.For neural network implementation, we utilized TIBCO Statistica data science platform which provides the ability to generate the neural network source code. This code was subsequently integrated into the implemented application written in Python.According to the experience of the maintenance department, the gradual seizure of the bearings is always accompanied by acoustic noise in the form of whistling. This was also confirmed during the regular inspections of the function parts condition of the carrier and assembly conveyor.By analysis of sound recordings, in the time and frequency domain, it looks for characteristic acoustic manifestations of bearing damage. This approach of early prediction does not allow identify specific bearing on which damage occurs. Therefore, this condition monitoring system works in the cooperation with the subsystem for measuring the temperature of individual bearings.An acoustic records of bearing failure, used as an example in this paper, were obtained from real operation on two carriers marked 25 and 44. These records were evaluated by the time domain statistical indicators and frequency analysis.When any bearing was damaged, a whistling sound was generated randomly at various carrier positions on the conveyor. However, the whistling always occurred when the carrier moved on the conveyor rail. All audio recordings were made at the same position on the assembly conveyor. The whistling sound is always recorded in them between 27 and 29 s, as highlighted in In time domain statistical scalar indicators which assess the signal amplitude are used for bearing diagnosis. As is known from the literature ,75, stanPeakRoot mean squareCrest factorKurtosisSkewnessTALAFTHIKATShape factori in considered time window, N is number of samples, and RMS0 is the root mean square value defined for a healthy bearing. After replacing of the damaged bearing with a new one, the first calculated value of RMS has been used as the RMS0 value in our solution.Impulse factorThe statistical descriptors, described above, were considered for analysis of acoustic signal parts of interest during which the whistle sound has occurred. Graphs in Generally, the conventional scalar indicators, such as the Peak and RMS, increase with smaller number of defects. When the bearing damage is moderate, the Kurtosis is the most sensitive indicator of the signal energy density. With the deterioration of the defects, the Impulse factor becomes the most sensitive indicator. The Shape factor is insensitive to damage spread. The TALAF parameter and THIKAT parameter describes in detail the evolution of the damage and achieves bigger difference between faulty bearing and bearing without failure. All observed statistical indicators decreased more or less significantly after the replacement of the damaged bearing with a new one.IF reduced after the bearing repair their size to about half the previous value.The changes of the scalar indicators of the damaged bearing on carrier 25 are presented in the line graphs in Due to these reasons, THIKAT and Kurtosis were selected as the assessing criteria of bearing state for time domain analysis in the proposed condition monitoring system.Due to the possible influence of the time analysis results by the assembly hall background noise from other sites near conveyor, the sound record of the carrier passage through the measurement place was also analyzed in the frequency domain.The fast Fourier Transform (FFT) transformation was utilized for digitalized audio signal analyzes in the frequency domain. The Hanning window function was applied on data blocks to zero-valued outside of the chosen interval and controlling leakage. The sample overlapping was not applied.One of the main symptoms of the bearing defect are ball-pass frequencies in the frequency spectrum. However, it is usually difficult to locate these frequencies in the spectrum if they are hidden in the background noise. Bearing defect characteristic frequencies are therefore calculated using following equations:Ball-passing frequency of the outer raceBall-passing frequency of the inner raceBall-spin frequency (about its axis)r is the relative frequency between the outer and inner race, BD is the rolling element diameter, PD is the pitch circle diameter, and α is the contact angle. The bearing frequencies of the spherical roller bearing type “SKF 22206 E”, for different speeds of the carrier, are calculated using Equations (15)–(18) and summarized in Fundamental train frequencyv is the speed of the carrier and frp is the over-rolling frequency of the damaged point on the rolling element (roller defect frequency).Where the parameter The typical frequency spectrum of audio signals, recorded at a measurement point for selected carriers 25 and 44, are shown in Identification of the bearing frequencies in the frequency spectrum of an acoustic signal is a very difficult due to small values being at the limit of FFT resolution. Due to the low carrier speed are the bearing frequencies (BPFO) localized in the range of lower microphone sensitivity, it is difficult to identify them in the frequency spectra. In The proposed algorithm for early condition monitoring of bearings state is based on the analyses of recorded sound signals in the time domain by calculating the selected descriptors. Using the frequency analysis, we detected the presence of magnitude increase for frequencies in range ±20 Hz around the whistle frequency. To identify the damaged bearing, the results of individual bearings temperature measurements have to be applied.Descriptors Kurtosis and THIKAT were used as auxiliary indicators to assess the condition of bearings by analyzing the audio recordings in the time domain. The magnitude value is tested in the frequency spectrum at a critical frequency of 1670 Hz. When the value of the Kurtosis indicator exceeds 5, the THIKAT value reaches 7 and the magnitude of the frequencies in the critical frequency range is greater than 100, the carrier is included in the group with a potentially damaged bearing. This can be further investigated by the maintenance personal to analyze the bearing state for future assembly conveyor modifications. This identification was also used to minimalize anomalies, which occurred in the implemented trigger-based bearing failure detection, using the carrier wheel temperature measurements.In our proof-of-concept phase, acoustic treating of the sound recording point was not possible and an alternative had to be found. Therefore, the magnitude limit value was chosen in a way that eliminates unwanted acoustic disturbances in the assembly hall.The proposed algorithm for the anomaly detection was implemented using the MathWorks MATLAB platform that was also used for the sound analyzes and provides all the required functionalities. These calculations were subsequently integrated into our application, written in the Python programming language. For MATLAB function executions, MATLAB Engine API available for Python was used. Although it is possible to rewrite the functions into the Python programming language, in our proof of concept we decided to utilize MATLAB functionalities directly, for easier integration with the sound analysis phases.To give the maintenance crew a tool for bearing health monitoring, a dashboard was implemented that allows them access to the collected data and anomaly detection results. The visualization was built on top of the ClickHouse database that integrated the data collected from the IIoT devices and stationary temperature measurements. For the identification and prediction of the bearing failure, we utilized the proposed algorithms that labeled identified anomalies in the collected data and created email and SMS notifications to notify the assembly hall maintenance. The example of the implemented dashboard is shown in The implemented dashboard shows carrier temperature measurements in the last two hours on the left side. Anomalies detected by the implemented trigger-based algorithms based on the thresholds and neural network together with the bearing health monitoring by sound measurements are shown on the upper right side. For the detected carrier, the temperature profile for the last seven carrier rotation is shown on the lower right side.The dashboard provides an overview of all the important monitored assembly carrier parameters required by the maintenance crew. Together with email and SMS notifications, it provides powerful tool for minimizing assembly conveyor downtime.The advantage of the IIoT data collection approach is in our case a complex coverage of the measurement of individual devices, during the entire assembly process, which provides opportunities for a more comprehensive analysis of the problem area. It is also suitable for collecting data from constantly moving devices, due to the fact that the operating time is several months or years without external power supply.A limitation in our case was the mounting of sensors on individual devices. Mechanical intervention on the carriers was not the preferred method of device and sensor mounting. Due to this limitation, the usage of vibrometer devices was not considered for anomaly detection. A complete implementation of IIoT sensors would also require the use of custom-built IIoT devices that would support six temperature sensors, but these are not available as standard.Despite the suitability of the IIoT data collection approach, it was unacceptable in terms of the plant’s production reliability strategy to install IIoT sensors on each carrier. For this reason, as an alternative solution, we proposed and implemented a stationary temperature and sound measurement infrastructure, which allowed us to measure and record the temperatures and sounds of all carriers in the assembly conveyor.With this approach, we created a universal, inexpensive, and easy-to-implement solution, where it is possible to measure values from any number of carriers and on any type of assembly line. The disadvantage of this solution is that the availability of the measured temperature information is only every two hours, when the carrier passes through the temperature measurement point on the carrier lift.From the point of view of bearing failure detection, the implemented solution using trigger-based anomaly detection seems to be a suitable approach. The main advantage was the possibility of a very fast deployment in this approach, given that from the available data it was possible to derive parameters for anomaly detection very soon requiring a small amount of data.Using this approach, we were able to detect an anomaly, a bearing failure, approximately 2 to 4 h before critical bearing damage during normal operation. A limitation of this solution is the need to manually set the detection parameters to capture only the actual bearing failure while minimizing false detection, which causes confusing notifications to the operator.Applying a neural network approach allowed us to eliminate the need to manually set parameters for anomaly detection. Based on the available data we were able to predict critical bearing damage using the proposed neural network, based on the currently measured bearing temperatures. This approach was able to predict bearing damage approximately 4 to 6 h before critical damage.To deploy this approach, it is necessary to have a sufficient amount of data available to learn the neural network, which makes it possible to deploy this approach later.Both previous approaches detect bearing damage indirectly due to increasing temperature caused by guide line friction.Therefore, we decided to supplement the anomaly detection with sound analysis, which is possible to predict the bearing state on the carrier in the phase when the limitation of its functionality begins. This can give the maintenance crew time to analyze bearing health before the failure itself and can lead to assembly line modification to mitigate the bearing failure causes. This means that with this approach it is possible to detect bearing state anomaly approximately 8 to 10 h before its critical damage.The disadvantage of this approach is the impossibility of accurately identifying a particular damage bearing on the carrier due to the fact that the sound is analyzed as a whole.Although the used methods are calculated and performed over the data store in the SQL storage, there should be no performance limitations implementing them on the used IPC directly performance wise. The anomaly detection, using a trigger-based approach and neural networks, can provide results in near real time. Detection using the sound analysis can provide results in a few seconds that is still a fraction of the time between the individual sample recordings.Each of these approaches can be used separately, but by appropriately combining these approaches, using their advantages and suppressing their disadvantages, we have proposed and implemented a unique approach that allows more accurate damage prediction and significantly minimizes the possibility of false detection.While the symbiosis of the neural network and sound analysis are used for prediction well in advance, the identification of damage by trigger-based anomaly detection is a direct starter of the maintenance process.Damaged carrier wheel bearing is the most common cause of cessation of the assembly conveyor. Using a proposed analysis of the sound signal, it is possible to predict the failure of a bearing on the carrier in the phase when the limitation of its functionality begins. Subsequently, with the deterioration of bearing damage, its temperature increases. This allows the identification of the specific wheel or guide bearing using our trigger-based detection and neural network approaches.The implemented smart monitoring system made it possible to predict and detect damage to the bearings of the transport carrier in time at a stage when the operation of the conveyor was not endangered. Alerts are sent to the maintenance department when bearing failure occurs using email and SMS notifications. Thus, a preventive replacement of a damaged bearing could be performed during the next regular scheduled production outage, minimizing production downtime.The maintenance crew thus received a powerful tool to predict and identify the damaged bearing before its complete seizure. The use of an implemented application platform can prevent an unexpected cessation of the entire assembly process and avoid significant financial losses. Performing the predictive maintenance, based on notifications from the realized application platform, ensures assembly process continuity and compliance with the planned number of produced cars. This is what our proposal has realized.


In the field of clinical ultrasound, the full digitalization of diagnostic equipment in the 2000s enabled the technological development of quantitative ultrasound (QUS), followed by multiple diagnostic technologies that have been put into practical use in recent years. In QUS, tissue characteristics are quantified and parameters are calculated by analyzing the radiofrequency (RF) echo signals returning to the transducer. However, the physical properties  of the biological tissues responsible for the imaging features and QUS parameters have not been sufficiently verified as there are various conditions for observing living tissue with ultrasound and inevitable discrepancies between theoretical and actual measurements. A major issue of QUS in clinical application is that the evaluation results depend on the acquisition conditions of the RF echo signal as the source of the image information, and also vary according to the model of the diagnostic device. In this paper, typical examples of QUS techniques for evaluating attenuation, speed of sound, amplitude envelope characteristics, and backscatter coefficient in living tissues are introduced. Exemplary basic research and clinical applications related to these technologies, and initiatives currently being undertaken to establish the QUS method as a true tissue characterization technology, are also discussed. Ultrasound has been widely used in medicine since its biological effect was confirmed by Langevin in 1917. Ultrasonic diagnostic equipment became widely available in the 1950s and is used in mainly qualitative diagnosis by the visual observation of signal waveforms in A-mode, tomographic images in B-mode, and dynamic images in M-mode. During the period when quantitative evaluation of blood flow became possible using the pulse Doppler method in the late 1960s, further quantitative diagnostic technologies that employed the physical characteristics of ultrasound waves as an index were proposed for development. The full digitalization of diagnostic equipment in the 2000s enabled the development of quantitative ultrasound (QUS) techniques, some of which have been put into practical use in recent years. In QUS, the radiofrequency (RF) echo signals returning to the transducer are analyzed, and parameters that can be used to quantify the tissue characteristics are calculated. In-phase/quadrature-phase (IQ) data are also used instead of RF data.In this paper, typical examples of QUS techniques for evaluating attenuation, speed of sound, amplitude envelope characteristics, and backscatter coefficient in living tissues are described, and examples are provided of exemplary basic research and clinical applications related to these technologies. Current work under way to establish QUS methods as a true tissue characterization technology are also discussed.Attenuation, defined as the loss of ultrasound signal energy with the propagation depth as a function of frequency, is essential in quantifying tissue properties. Ultrasonic waves that are emitted from the probe and pass through living tissue are reflected and scattered by scatterers smaller than the ultrasonic pulse, pass through mutual interference, and are received by the probe as RF echo signals. The attenuation characteristics of the target tissue can be evaluated by evaluating the backscattered signal that has returned in the same direction as the transmission. The accuracy of the evaluation is dependent on the state of scatterers in the living tissues , as well as the shape and resolution of the transmitted and received ultrasonic beams.The attenuation evaluation techniques proposed by numerous researchers over the past decades can be broadly divided into the time domain approach and the frequency domain approach. However, because time and frequency are closely related in echo signals, some techniques use them in tandem. Although the time domain approach is simple and easy to implement, it is inferior in robustness, and accordingly, the frequency domain approach is often used in reality. The attenuation coefficient is the main index used for quantifying attenuation. It should be noted that the unit of the attenuation coefficient in the medical ultrasound field is generally dB/cm/MHz, but dB/m is sometimes used.The spectral difference method is the most basic technique for evaluating the reduction in the echo signal power spectra along the propagation path of the ultrasound beam, and it has a long development history. Kuk proposed the basic theory of attenuation measurement using broadband pulses in 1978 , and comThe spectral shift method is the most common technique for evaluating the downshift of the echo signal power spectra of an ultrasound beam propagating through living tissues. This algorithm uses the downshift in center frequency of the power spectrum versus propagation depth to estimate the attenuation slope. Local attenuation is evaluated from the attenuation slope of the echo signal in the ROI being evaluated. Regarding the basic methods for determining the attenuation coefficient of tissues, the tissue is generally assumed to have linear frequency-dependent attenuation. In reality, however, many tissues exhibit non-linear frequency-dependent attenuation. Ophir investigated the relationship between downshift of the center frequency of the spectrum and the attenuation coefficient  when it is assumed that living tissue with non-linear characteristics is linear, and proposed an in vivo measurement technique with narrow band pulses , 6. Kim In general, spectral shift methods are more robust than the spectral difference methods at the boundary region of backscatter changes; however, they each have specific limitations. Classical spectral shift approaches for estimating ultrasonic attenuation are more sensitive to local spectral noise artifacts and have difficulty in compensating for diffraction effects due to beam focusing. In contrast, spectral difference approaches fail to accurately estimate attenuation coefficient values at tissue boundaries that also possess backscatter variations . Kim proEvaluation of attenuation based on the frequency shift of the received signal.Evaluation of attenuation by comparison with a phantom in which the attenuation and scattering coefficients are known.Evaluation of attenuation by comparison with training data obtained with known transmitting and receiving conditions.Evaluation of attenuation based on the slope of the ratio of signals transmitted and received at two different frequencies.The high functionality of modern ultrasonic diagnostic equipment has led to the practical application of high-frequency and high-resolution attenuation evaluation methods. Dedicated diagnostic ultrasonic units have been produced for each of the spectral difference, spectral shift, and hybrid methods. The following four types of applications are in current use.Of course, even if the same theory is used, differences exist such as filtering and the combination of several methods depending on the manufacturer and diagnostic equipment. There has been particular interest in the application of attenuation evaluation methods to the field of gastroenterology, in which the results of numerous studies have been reported in the past few years –19. AtteWhen ultrasound waves propagate through the living tissue under observation, the speed of sound is an important parameter that indicates the acoustic characteristics peculiar to a particular living tissue, and it greatly affects the imaging of echo signals in B-mode or M-mode. In general, ultrasonic diagnostic equipment assumes that living tissue is a uniform medium, and it sets the reference speed of sound at 37 °C as 1530 m/s according to the Japanese Industrial Standard (JIS) [1540 m/s according to the American Institute of Ultrasound in Medicine (AIUM)]. However, deviation of the propagation path due to the complexity of the actual tissue structure in living tissue and refraction are not taken into consideration. Therefore, distance measurements based on the received echo signals and the imaged tomographic image will contain errors and distortions. These are considered to lie within the acceptable error range in practical use. However, as the frequencies of ultrasound waves used in diagnosis (and applied to QUS) are continuing to increase, it is necessary to estimate the speed of sound in the local region with greater accuracy. Accurate evaluation of the local speed of sound is also beneficial in calculation of the attenuation coefficient. The following section outlines the techniques for improving image quality by evaluating the speed of sound from the RF echo signal and using differences in the speed of sound at each local region.In conventional ultrasound diagnostic equipment, an assumed average speed of sound is used for delay and sum beamforming to create a B-mode image that is generated by the RF signal of each received echo line. Conversely, as proposed by Ogawa and Umemura  and HayaIn the spatial coherence method, spatial coherence is calculated under various conditions, taking into account the number of elements in the probe used for receiving the echo signals, and the speed of sound in a local or wide area is evaluated –30. SincIn the compounding method, the speed of sound is evaluated directly by searching for spatial shifts in images between different transmission and reception angles. Specifically, the discrepancy between the speed of sound used for beamforming and the speed of sound of actual living tissue is used as a reference. If the speed of sound mismatch is high, a spatial error occurs, because the optical path length changes when different angles are used. When the magnitude and direction of the shift are optimized, the difference between the assumed speed of sound and the actual speed of sound is minimized –43. The Technologies for evaluation of the speed of sound  have also been implemented in clinical ultrasound diagnostic equipment, and various applications have been reported. Hirooka has reported the clinical applications of this technology, since the early stages of its development . ImbaultIn actual living tissues, signals from a small target tissue can become buried by scattered signals from the minute scatterers that are often randomly and densely present over a wide area of surrounding tissues in the medium. In a situation where there are 10 or more scatterers in the resolution cell, which is the resolution of the ultrasound beam, an extremely weak scattering signal is generated in each microscatterer, and received by the probe. The received signal includes a noise signal produced as the result of their interference. Accordingly, a speckle pattern is observed in the final B-mode image. The size of the mottled speckle pattern is determined by the sound field characteristics of the irradiated ultrasound, and there is no correlation between the structure of scatterers in the living tissue and the speckle pattern. In other words, if the observation area contains speckle, then the tissue is in a dense and homogeneous state. Many researchers have proposed the use of probability density functions to express the properties of RF echo signals that exhibit a speckle pattern. Amplitude envelope statistics is a QUS method that normalizes the amplitude envelope characteristics of RF (or IQ) echo signals with the probability density functions. This technology has a long history with ultrasonics in medicine.The Rayleigh distribution, which is a probability density function , has beeThe Rice distribution, which was proposed by Nakagami  and by RThe QUS technique based on the Rayleigh distribution has already been implemented in clinical equipment, primarily for the purpose of assessing liver fibrosis , 71–73, K-distribution and homodyned K-distribution are of particular value in evaluating tumors. In a pioneering study, Shankar developed a method of breast tumor classification for clinical application , 54 usinCurrently, basic studies that use clinical data most commonly employ the Nakagami model. A wide variety of targets have been evaluated in such studies, including vascular studies by Huang , ophthalThe backscatter coefficient (BSC) is defined as the time-averaged scattered intensity in the backward direction per unit solid angle per unit volume, normalized by the time-averaged incident intensity. General B-mode generation and amplitude envelope statistics mainly use time information. However, backscattered signals are converted into the frequency domain for analysis in BSC estimation. BSC is frequency dependent, because ultrasonic scattering is affected by the intrinsic acoustic impedance and size of the scattering medium. In other words, BSC evaluation is a key method for estimating microstructural characteristics such as the shape, size, composition, and concentration of the tissue to be observed, as well as the impedance ratio between the scatterer and the surrounding medium.Sigelmann and Reid first developed a method for estimating backscatter power from a volume of randomly distributed scatterers using a single-element planar transducer . SubsequBSC-based QUS has been applied to characterizing the tissue microstructure of the liver, prostate, pancreas, spleen, eye, and lymph nodes, among others , 99–103.Franceschini also proposed a method for calculating the BSC under arbitrary conditions in which the actual structure of living tissue is prepared and the acoustic impedance of each tissue is presented as a two-dimensional or three-dimensional computer model , 107. ThFigure Several QUS methods (including shear wave elastography and transient elastography) that use the acoustic properties of living tissues as evaluation parameters have been proposed and realized, as described in the present review. However, the relationships between the types of physical properties  of the biological tissue responsible for the image features and QUS parameters have not been sufficiently verified because of the variety of conditions under which living tissue is observed with ultrasound. In addition, discrepancies always exist between theoretical and actual measurements. A major problem in the clinical application of QUS is the dependency of the evaluation result on the acquisition conditions of the RF echo signal, which is the source of the image information and varies among diagnostic equipment. Dependency on the acquisition conditions and equipment has been a longstanding concern in medical imaging modalities such as CT and MRI, as well as in ultrasound. However, QUS studies have most commonly been conducted by independent researchers, and a comprehensive study has not been completed.The Quantitative Imaging Biomarkers Alliance (QIBA)  was estaTo verify the accuracy of QUS methods for evaluating the properties of echo signals, it is necessary to understand the intrinsic acoustic characteristics of individual living tissues. Ultrasonic observation with a higher resolution than is possible at the clinical level is indispensable for this purpose, and is realized by scanning acoustic microscopy (SAM). SAM uses ultrasound frequencies of 100 MHz or higher. Spatial resolutions of 15 and 1.5 µm are obtained at frequencies of 100 MHz and 1 GHz, respectively, and it is possible to observe organelles at frequencies above 200–300 MHz.Liquid media or unstained tissue specimens sliced to 4–20 µm are commonly used to assess attenuation and speed of sound –118. SAMIn recent years, in addition to attenuation, speed of sound, and acoustic impedance, multiple indicators such as thickness, density, and bulk modulus have also been compared , 124. AcVarious QUS methods, particularly shear wave elastography, attenuation evaluation, and speed of sound evaluation have been developed and implemented in clinical equipment and are now being applied in a wide range of fields. The theory behind and full meaning of the QUS parameters produced by such equipment are not fully understood at present; however, large-scale projects  are under way to address this problem, and working group activities are being actively promoted in the participating academic societies. Basic research to support theory and practice is also continuing to develop. It is important that users of the QUS method not only use the technology but also recognize that their research results will be added to large-scale standardization studies that are currently under way. In addition, considering the wide diversity of QUS methods, it is essential to conduct a thorough investigation into the suitability (or unsuitability) of each QUS method for the particular area to be evaluated. Despite current challenges, because QUS technology incorporates the properties and advantages of ultrasound, it undoubtedly has great potential for future application.