Pharyngeal fricative is one typical compensatory articulation error of cleft palate speech. It passively influences daily communication for people who suffer from it. The automatic detection of pharyngeal fricatives in cleft palate speech can provide information for clinical doctors and speech-language pathologists to aid in diagnosis.This paper proposes two features  to detect pharyngeal fricative speech. CSIFs feature is proposed to detect the distribution characteristics of frequency components in pharyngeal fricative speech caused by the changed place of articulation and movement of articulators. While OSPP is presented to reflect the concentration degree of prominent peak which is closely related to the place of articulation in pharyngeal fricative, both features are investigated to relate to the altered production process of pharyngeal fricative. To evaluate the capability of these two features to detect pharyngeal fricative, we collected a speech database covering all the types of initial consonants in which pharyngeal fricatives occur. In this detection task, the classifier used to discriminate pharyngeal fricative speech and normal speech is based on ensemble learning.The detection accuracy obtained with CSIFs and OSPP features ranges from 83.5 to 84.5% and from 85 to 87%, respectively. When these two features are combined, the detection accuracy for pharyngeal fricative speech ranges from 88 to 89%, with an AUC (area under the receiver operating characteristic curve) value of 93%. Cleft palate is one of the most common craniofacial alterations and birth defects . It reprPharyngeal fricatives are produced when the base of the tongue approaches the posterior pharyngeal wall , 5. ThisIn clinic, three classes of approaches are generally used for the diagnosis of pharyngeal fricatives. (1) Medical instruments: clinicians use invasive instruments such as endoscopes , 11, latWith the aim of relieving the pain of patients with pharyngeal fricative and exploring more efficient approaches, many studies have been carried out based on speech signal processing in recent years. Speech signal processing is a noninvasive technique based on digital signal processing, which is an efficient tool for the objective diagnosis of speech disorders . In the In this paper, we aim to achieve automatic detection of pharyngeal fricative speech. The six types of initial consonants in pharyngeal fricative speech are analyzed initially. The generation of initial consonant is a dynamic process . The comThis paper aims to achieve automatic detection of pharyngeal fricative speech. CSIFs and OSPP features are proposed to reflect the changes in the production process of pharyngeal fricative speech. In this section, the significance test for both proposed features and the experiments of pharyngeal fricative speech detection using these two features are conducted.This section is composed of five parts. In the first part, the classifier used to evaluate the performance of the proposed features is described. In this classifier, several weak classifiers are trained to form a strong classifier. In the second and third part, the results of CSIFs and OSPP features calculated from normal speech and pharyngeal fricative speech are described and analyzed. In the fourth and fifth parts, the results of significance test for these two features and the detection results of pharyngeal fricative speech using the features are given and discussed.In our research, the pharyngeal fricative speech detection task is performed by applying the bagging of the ensemble learning classifier. The idea of ensemble learning is principally based on the theoretical cornerstone that the generalization ability of an ensemble is usually much stronger than that of a single learner –34. Theri = 1, 2…, T), which are formed by bootstrap sampling on the original database. The final ensemble classifier H predicates the label of speech signals by majority voting of each base classifier. The output class has more votes by these base classifiers. In this work, each initial consonant is segmented into N frames. The proposed CSIFs and OSPP features are calculated based on the frames. Then, there are N or 2*N feature values that represent each initial consonant. They are input into this classifier as training parameters. This process is illustrated by Fig. In this work, this classifier is a bootstrap-aggregated ensemble of decision trees. The decision tree algorithm includes three types of nodes, the root node, internal nodes, and end-note or target . This trmentclass2pt{minimThis classifier is applied to evaluate the performance of automatic pharyngeal fricative detection system using CSIFs and OSPP features based on k-fold cross-validation (k-CV) algorithm. The k-CV algorithm is commonly used in the field of machine learning –41. In tIn this subsection, the CSIFs values of the six types of initial consonants are presented to observe their distribution characteristics in normal speech and pharyngeal fricative speech. These results for each type of initial consonant are shown in Fig. In Fig. The calculation of CSIFs is based on the relation between self-defined HER and LER. The HER and LER are defined considering the amplitude distribution of frequency components in the speech spectrum of pharyngeal fricative speech and normal speech. They are closely related to the production process of speech signals. The difference in the distribution range of CSIFs values in pharyngeal fricative speech and normal speech is also caused by the different production processes. Speakers with pharyngeal fricative tend to contract pharyngeal muscle and make the posterior displacement of tongue to form a pinch point . MeanwhiIn this part, the OSPP values of the six types of initial consonants are presented to analyze the overall distribution of the prominent peaks in normal speech and pharyngeal fricative speech using the boxplot. The OSPP values of initial consonants /c/, /ch/, /q/, /s/, /sh/, and /x/ are illustrated in Fig. Figure The difference in the distribution range of OSPP values between normal speech and pharyngeal fricative speech is mainly caused by the change in the place of articulation. Speakers suffering from pharyngeal fricative are accustomed to changing the place of articulation to keep the manner of articulation . Their pH = 1:There are significant differences in this feature between pharyngeal fricative speech and normal speechH = 0:There are no significant differences in this feature between pharyngeal fricative speech and normal speechSignificance testing is a common technique in statistics to evaluate the significant difference in several data sets . It has p value (probability of test error occurring), and the result of the hypothesis.There are six types of initial consonants in our research. The significant test results are listed in Table p-value is much less than 0.01. These parameters indicate that CSIFs and OSPP features have significant differences between pharyngeal fricative speech and normal speech for the six types of initial consonants.As shown in Table In this research, CSIFs and OSPP are proposed to detect pharyngeal fricative in cleft palate speech. In this subsection, the results of automatic pharyngeal fricative speech detection using CSIFs and OSPP features are presented and discussed. The performances of these two features are evaluated by the ensemble learning classifier. The general performances of these two proposed features are discussed in terms of accuracy, sensitivity, specificity, and AUC. Sensitivity reflects the probability of missed diagnosis in clinical, while specificity reflects the probability of misdiagnosis. AUC is the area under the receiver operations characteristic curve. It is given to show the results more compactly . It can In this experiment, CSIFs and OSPP features are used to form different feature sets to discriminate pharyngeal fricative speech and normal speech. The experimental results of automatic detection of pharyngeal fricative speech are listed in Table As shown in Table In pharyngeal fricative speech, the movements of articulators and the places of articulation are different from those in the production process of normal speech. Since the movements of articulators in the oral cavity of the pharyngeal fricative decrease and the places of articulation change, it is aware that the speech spectrum of pharyngeal fricative speech is modified in a wider spectral range with lower amplitudes. The main objective for CSIFs is to reflect this modification. In the calculation of CSIFs, HER and LER are defined in each initial consonant, considering the modified spectral range in pharyngeal fricative speech. Then, speech signals in the HER and LER are extracted. CSIFs can reflect this distribution difference of frequency components between pharyngeal fricative speech and normal speech by evaluating the relation of signals in HER and LER. For the OSPP feature, the prominent peak of the energy-concentration region is mainly influenced by the place of articulation. Since the posterior displacement of the tongue occurs in the production of pharyngeal fricative, its prominent peak may shift to a lower frequency part than normal speech, and the amplitude of the peak decreases. The OSPP feature focuses on characterizing the most prominent peak in the speech spectrum. The results of CSIFs and OSPP features indicate that they are useful to represent the changes occurring in the pharyngeal fricative speech. The proposal of these two features is based on the change in the pronunciation process of pharyngeal fricative. When OSPP and CSIFs features are combined to detect pharyngeal fricative, the detection accuracy increases. The discrimination between pharyngeal fricative speech and normal speech is improved. This improvement is probably caused by the combination of these two features. The information on the changes in place of articulation and movement of articulators is included in the same feature set.To observe the classification performances of pharyngeal fricative detection on different types of initial consonants, the detection experiments of pharyngeal fricatives on each type of initial consonant are conducted. The experimental results are listed in Table As listed in Table In this subsection, the comparative experiments are conducted based on features proposed in previous researches –26 and tAs shown in Table As shown in Fig. Compared with the work of Xiao et al. , 24 and Compared with the work of Fu et al. , PP 26]26] is thThese features in previous work –26 are tIn our work, the main objective is to achieve automatic detection of pharyngeal fricative speech. CSIFs and OSPP features are primarily proposed based on the production process of pharyngeal fricative in cleft palate speech. Both features are useful for analyzing the place of articulation and movement of articulators in the generation process of speech signals. In this paper, their capabilities to discriminate pharyngeal fricative speech and normal speech are tested and discussed. This test is conducted on collected speech database, which consists of 1208 speech samples including six types of initial consonants. Moreover, the comparison experiments with previous studies are also conducted.The proposed CSIFs feature aims to reflect the distribution differences of frequency components between pharyngeal fricative speech and normal speech. Since pharyngeal fricative speech has more posterior place of articulation and fewer movements of articulators in the oral cavity, its energy tends to shift to a lower frequency part than that of normal speech. Correspondingly, its low-energy region has a wider frequency range. CSIFs feature detects this distribution characteristic by evaluating the correlation of signals in high-energy region and low-energy region defined in this work. The experimental results of CSIFs note that CSIFs values calculated from normal speech are mostly lower than those calculated from pharyngeal fricative speech. The detection accuracies of pharyngeal fricative speech using CSIFs feature are above 83.5%. These results indicate that it has the capability to discriminate pharyngeal fricative speech and normal speech. In addition, the matching filter is designed to locate the high-energy region and low-energy region of each initial consonant in the calculation process of CSIFs.In pharyngeal fricative speech, the posterior displacement of the tongue occurs in its generation process. This posterior displacement influences the energy-concentration regions in the spectrum. Generally, it results in lower amplitudes and center frequencies in the energy-concentration regions of pharyngeal fricative speech. The OSPP feature detects the change in the place of articulation in pharyngeal fricative speech by evaluating the most prominent peak of the energy-concentration regions in the spectrum. The detection accuracies of pharyngeal fricative speech obtained with OSPP feature range from 85 to 87%. These results indicate that OSPP feature has good capability to detect pharyngeal fricative speech. When the OSPP feature and CSIFs feature are combined in the same feature set, the best result is obtained with an accuracy of 89%. This result notes that the combination of OSPP and CSIFs features can improve the capability to discriminate pharyngeal fricative speech and normal speech.A comparison with related researches on pharyngeal fricative speech detection is also conducted in this paper. Xiao Y et al. , 24, He Finally, the proposed features could be adopted in other speech diseases and other languages, since there exist many similarities in the place of articulation and the place of manner between Chinese and other languages. A new choice is provided for the doctors and patients. In any choice, an informed selection of the techniques that are applied to model the speech signal could help the speech therapist and the clinician to make more accurate decisions regarding the pharyngeal fricative and the treatment prescription for the patients.This section describes the details of the material and the proposed methods. In the first part, the details of the participants and collected speech samples are introduced. In the second part, the pronunciation process of pharyngeal fricative is described. In the following two parts, all processes of the proposed features to discriminate pharyngeal fricative speech and normal speech are clearly described, including their proposals, objectives, and detailed calculation steps.Fifty patients  age: 20.3 ± 8.1 years) with pharyngeal fricative and 50 healthy controls  age: 19.2 ± 7.8 years) participated in this research. All speech recordings are collected from the West China Hospital of Stomatology, Sichuan University. It is a prestigious and well-known medical center in China and it has the largest number of cleft palate patients in the world. This study is approved by the West China Hospital of Stomatology Institutional Review Board (WCHSIRB-CT-2013-011). All participants sign their informed consent prior to their inclusion in this research.In the research on pharyngeal fricative speech detection, one bottleneck is the collection of speech samples. The sample size depends on the types of collected initial consonants and the amount of enrolled patients. Considering the types of initial consonants, there are six types of initial consonants  in which pharyngeal fricative may occur. However, in previous studies –26, onlyThese speech samples are collected in a specific studio and sampled with 44.1 kHz sampling rate at a resolution of 16 bits in the hospital. These speech recordings are independently rated by three SLPs. Each recording is selected for our database only if all three speech-language pathologists give the same assessment of it.The production of speech is a complex physiological process. This physiological process culminates in the movement of articulators to control the airflow. In the production process of normal speech, the air flows from the lungs and passes through the glottis, while if the airflow is impeded by glottis adduction, a voiced sound is produced. Next, airflow enters the upper vocal tract, where articulators interact to control the airflow . With thPharyngeal fricative, a typical compensatory articulation error in cleft palate speech, is caused by abnormal pronunciation habits. These habits arise to compensate for velopharyngeal insufficiency . Due to ρ and u are the density and speed of the airflow, respectively; D represents the size of vocal tract. The size of the human vocal tract is different for various places and manners of articulation in different pronunciations. When the Reynolds number in the place of articulation is larger than 2100, turbulent noise is generated [Speech signal generation is a complex nonlinear process with the movements of articulators. In this process, if airflow in the lungs goes through a narrow passage in the vocal tract, turbulent noise may be generated. The generation of turbulent noise depends on the Reynolds number, which relies heavily on the size of the vocal tract and the velocity of the airflow , 46. Theenerated .Speakers suffering from pharyngeal fricative attempt to change their place of articulation to compensate for velopharyngeal insufficiency. The changed place of articulation is the pharynx, with the root of the tongue against the back of the pharynx. In this place of articulation, the gap between the pharynx and tongue root decreases, and the velocity of airflow increases. According to , the ReyThe change in the place of articulation in the production process of pharyngeal fricative results in reduced quality and intelligibility of generated speech signals. Speech quality and intelligibility are closely related to the amplitude variations of frequency components and their distribution. . These iAs shown in Fig. Speakers with pharyngeal fricative restrict airflow in the pharynx by posterior displacement of the tongue . This poAs shown in Fig. In this work, the cut-off frequency of HER and LER is calculated based on the amplitude variations in the Fourier spectrum. However, the amplitude variations in the spectrum of each initial consonant differ in terms of various characteristics, such as vocal tract length, gender, age, place of articulation, and manner of articulation. Thus, the cut-off frequency of each initial consonant is different. To calculate the cut-off frequency, a matching filter is designed to realize self-adaptive calculation for each type of initial consonant.This calculation process of the designed matching filter can be summarized as follows. First, the Fourier spectrum of each initial consonant is calculated and segmented into short segments. Second, the variance envelope is computed from the segmented spectrum. Then, linear regression fitting is utilized to locate the cut-off frequency of HER and LER. The detailed calculation process of cut-off frequency is described as follows:The Fourier spectrum of each initial consonant is calculated by fast Fourier transform (FFT size = 1024). Then, the spectrum is segmented into short segments using the Hamming window. The selection of the window size depends on the sampling frequency of speech signals. In this work, the sampling frequency of speech signal is 44,100 Hz. A window size is selected as 689 Hz, while the window shift is 344 Hz.v = 1, 2…, N, where N is the number of segments) is composed of variance values that are calculated from spectrum segments. For each spectrum segment, its variance is calculated by  is the ith data point in the current spectrum segment. M represents the variance of the current spectrum segment, Variance envelope entclass1pt{minimaAs shown in Fig. cth variance c = 1, 2…, N, where N is the length of variance envelope) to the last variance In variance envelope  fitting  process After the processing of the linear regression fitting algorithm in Algorithm 1, the slope matrix that contains all the slope values of the fitted linear regression lines is calculated. The normalized variance envelope and slope values are shown in Fig. R, while the solid line is the variance envelope. The slope matrix R magnifies the difference between the HER and LER in the speech spectrum of the initial consonants.As shown in Fig. R, slope value c = 4, 5…, N-2, where N is the length of matrix R) is compared with In slope matrix c_f is the calculated cut-off frequency. Coor is composed of all frequency index of all spectrum segments. H denotes the results (0 or 1) of the significant test [A method is presented in the comparison process shown in Fig. ant test . When H c_f is obtained by calculating the mean value of frequencies in cth spectrum segment. The c_f is the cut-off frequency between LER and HER region.In steps 5–7, when H, slope The cut-off frequency calculated above is adopted in Butterworth high-pass and low-pass filters. Thus, the HER and LER of an initial consonant are obtained. The order of these two Butterworth filters is set as 10. This order is suitable for the filtering of speech signal . To haveIn these two equations, mentclasspt{minimaAs shown in Fig. The Pearson correlation coefficient (PCC) is used to calculate CSIFs in this work. PCC  can evalN) is the frame index of the signals in HER and LER. The frames with the same frame index of HER and LER are as one group to calculate PCC. For the nth group of frames  is calculated using (5).nth frame of the HER, nth frame of the LER. E(*) is the expectation. Using . It is performed frame by frame to calculate this spectrum.fc, and f2. The bandwidth of each frequency band equals (f2 − f1). The relationship of these three frequency parameters is described by \documenti is the frequency band index) is the mean value of the amplitudes in the frequency band. Then, the one-third octave spectra on the band level are represented by E.Spectral line calculation: for each extracted frequency band of the 43 frequency bands, its spectral line The calculation of one-third octave spectra is divided into three steps:Ei using the least square method [The linear regression line is calculated by fitting the frequency band index of one-third octave spectrum to its corresponding e method . Since te method , linear DF is the differential value of each frame, n  is the frame index, peak represents the maximum of the most prominent peak in one-third octave spectra, RL denotes the value of the linear regression line with the same horizontal ordinate of the prominent peak.The differential value is calculated from the most prominent peak value of one-third octave spectra and the corresponding value on the regression line that is directly below the peak. The OSPP feature of each initial consonant is formed by the group of each differential value. This calculation is described in (9).An example of the one-third octave spectra and its linear regression fitting line of normal initial consonant /s/ is given in Fig.


The digitization of manufacturing industry has led to leaner and more efficient production, under the Industry 4.0 concept. Nowadays, datasets collected from shop floor assets and information technology (IT) systems are used in data-driven analytics efforts to support more informed business intelligence decisions. However, these results are currently only used in isolated and dispersed parts of the production process. At the same time, full integration of artificial intelligence (AI) in all parts of manufacturing systems is currently lacking. In this context, the goal of this manuscript is to present a more holistic integration of AI by promoting collaboration. To this end, collaboration is understood as a multi-dimensional conceptual term that covers all important enablers for AI adoption in manufacturing contexts and is promoted in terms of business intelligence optimization, human-in-the-loop and secure federation across manufacturing sites. To address these challenges, the proposed architectural approach builds on three technical pillars: (1) components that extend the functionality of the existing layers in the Reference Architectural Model for Industry 4.0; (2) definition of new layers for collaboration by means of human-in-the-loop and federation; (3) security concerns with AI-powered mechanisms. In addition, system implementation aspects are discussed and potential applications in industrial environments, as well as business impacts, are presented. The adoption of innovative digital technologies, also referred to as Industry 4.0, is progressively leading to improved product quality, work safety, fault predictions and efficiency in energy use and production ,2,3. IndThe industrial internet of things (IIoT) is a physical network of things, objects or devices (that contain embedded technology) for sensing and remote control, in an industrial context, that allows greater integration between physical and cyber worlds . In the However, the adoption and integration of AI-based innovation in the manufacturing domain comes with a few hurdles and caveats that must be properly addressed in order to take advantage of its full potential, without jeopardizing the irreplaceable role of humans and the protection of sensitive data and procedures ,23. In tThe rest of the manuscript is organized as follows. In The motivation behind this work is the need to provide an IIoT-based system that increases performance and safety in the manufacturing domain. This system, which enables the digitization of the relevant processes in critical infrastructures like manufacturing plants, should be secure, reliable, resilient and privacy-preserving. At the same time, it should facilitate the interaction between humans and machines and support the collaboration between peer systems. In this context, AI has the potential to support several of these requirements in an efficient manner and, thus, it can be embedded into the fabric of a trustworthy IIoT platform.Today, AI solutions are implemented in dispersed and isolated components of manufacturing IT systems. In most cases, the successful application of AI consists of pilots or testbeds to optimize specific processes . To realize the full innovation potential of AI, it must be regarded as a cross-cutting concern for manufacturing IT systems. Current Industry 4.0 reference architectures do not properly integrate the needed building blocks such as new deployment paradigms , scalable data-processing pipelines and information models, and AI-enabled digital twins used for monitoring and optimizing business intelligence ,25,26. IAI technologies should not only be used for data analytics in support of business intelligence, but also for automated decision making on manufacturing process parameters and configurations. Moreover, the rise of the “as-a-service” model means that on-site IT systems increasingly communicate with third-party cloud-hosted services. Unauthorized and malicious access to these systems may have not only a significant economic impact but also may compromise the physical integrity of human workers on the shop floor. Faulty behavior may also be an unintended result of wrong design, because AI algorithms are often black-box models , while the inner workings of an algorithm fetched from a remote repository are not fully understood or the decisions of one algorithm create a conflict with other algorithmic decisions. Proper secure federation mechanisms and AI-based cyberattack risk analysis are crucial cross-cutting concerns in AI-based manufacturing systems. In addition, the performance of modern AI techniques requires large volumes of high-quality data which are often not available inside a single enterprise. From a business perspective, sharing data with other economic actors or even between different facilities of the same organization is often not feasible for reasons of Intellectual Property Rights (IPR) protection, General Data Protection Regulation (GDPR), cybersecurity, etc.Finally, in recent years, machines in the manufacturing sector have become specialized agents that autonomously perform a given task at an outstanding quality standard with a high level of repeatability. Industry 4.0 will make machines increasingly smarter by using AI models . In the With the objective to leverage unique human intellectual capabilities and to confirm and reinforce the role of human workers in an increasingly digitalized and automated environment, new intelligent design and decision-making tools must be developed to promote human agency and oversight, simplifying the understanding and usage of AI results and considering multiple collaboration schemes depending on the situation. Human-AI will work in tandem in any phase of the product construction process, from design, over intelligent manufacturing execution monitoring to predictive maintenance.The RAMI 4.0  provides a blueprint of manufacturing systems and has been developed to create a common perspective and develop a common understanding between all the actors involved. Structured along with well-defined layers , RAMI 4.0 provides a service-oriented reference architecture that spans all elements and information technology (IT) components (from product up to the connected world) along the whole product life cycle (from design to maintenance). To address the aforementioned challenges related to AI introduction in smart manufacturing, we propose three types of extension to the RAMI 4.0 architecture, as depicted in First, enhancement of all existing RAMI 4.0 layers, reflecting that AI is a cross-cutting concern affecting all functionality in IT manufacturing systems. Indeed, AI-augmented manufacturing requires novel forms of data processing and information modelling and AI will allow for more autonomous and automated business intelligence. Second, we define a human-in-the-loop layer that provides methods, models and tools to facilitate the collaboration among human and AI-based virtual entities within a manufacturing site, thus assisting humans to take the optimal decisions. Third, a federation layer is defined that promotes the exchange of knowledge in terms of training models, AI algorithms, threat analysis results, deployment recipes and best practices across manufacturing sites by integrating novel concepts, such as secure federated learning and AI-on-demand schemes.Finally, one of the main challenges towards the implementation and adoption of AI concepts in IT manufacturing systems is the increased need for interoperability at different levels of the manufacturing ecosystem . In thisIn this context, our architectural approach relies on cloud-native tools for timely data collection, processing and curation, relying on the dynamic instantiation of data pipelines, while addressing security, privacy and confidentiality concerns across the physical and virtual entities. In this way, our approach transforms the collected data and information models into AI-enabled functional intelligence, leading to business knowledge, actionable insights and informed decisions, while being capable of recognizing complex events and process deviations that cannot be easily and timely captured with human judgment. Also, this ensures that our system is scalable to support manufacturers, ranging from Small or Medium Enterprises (SMEs) with limited on-premises resources, up to large industries with private clouds.In compliance with the RAMI 4.0, our proposed service-oriented architecture consists of the layers depicted in Besides extensions to the existing Business, Functional, Information and Communication Layers, the architecture contains additional layers for hardware-in-the-loop (HITL) collaboration and for collaboration through federation with other manufacturing sites, while security is a cross-cutting concern. In the following, we describe the functionality of each component as shown in The main responsibility of the Communication and Information Intelligence layers is to perform all the needed actions on the factory-wide datasets so that the components of the upper layers can make decisions based on the outcome of AI algorithms running on top of such processed data streams or batch datasets. Raw (non-labeled) data generated by manufacturing devices are processed along with all other relevant info  of the lower layer and properly transformed before passing to the upper layers of our architectural approach. The goal is to have an enriched metadata library covering various and diverse data creating during manufacturing procedure.This layer also contains core functionality to deploy AI algorithms closer to the sensor (edge computing) and to detect shifts in the dataset statistics, indicating a need to retrain algorithms. In support of the two major categories of AI algorithms  the component should allow the possibility to set up pipelines consisting of typical data processing tasks  and AI models (used in the services of the upper layers). From this perspective, the component provides the mechanisms to create these pipelines easily as well as a set of out-of-the-box and extensible data processors to make the data handling easier and effective for the most common data used in the factory; (ii) the component should allow the deployment of the pipelines and the orchestration of the different components and frameworks used, including the AI models developed and made available via containers. To do so, the component should be able to automate the deployment process on distributed infrastructure  and to orchestrate the different modules and exact frameworks needed to run the processes.The lifelong edge-based learning component provides support for building deep-learning models that must be deployed on edge devices as part of data-processing pipelines ,35. EdgeThe intra-manufacturing knowledge graph serves as the central point for knowledge management in the platform. This module combines and extends currently available models, both domain-specific and domain-independent, to enable knowledge representation and linkage. Moreover, it applies and extends state-of-the-art reasoning and graph analytics algorithms for link prediction and entity consolidation to discover relationships across the information coming from the different layers and modules.The threat intelligence manager takes advantage of the collected and curated datasets and applies AI algorithms for executing threat analysis in order not only to predict potential cybersecurity incidents but most importantly to manage and mitigate such incidents in a timely manner. This component provides a solution that addresses the shortcomings of current signature-based methods  and anomaly-based methods .This layer accommodates all services and components modeling the status and behavior of all functions and assets (including humans) of the manufacturing process. These models are built using AI algorithms and trained models, leveraging the data processing capabilities and information models from the lower layers. On top of these, other AI-enhanced services will be built, either for automated business goal optimization or in processes with HITL. In the following, the components belonging to this layer are presented.The Component-oriented behavior models are digital twins with behavioral models of individual components; in this way, they are moving away from system-wide behavioral models currently used. Each component not only digitally represents the state of assets and properties through the asset administration shell  but also models the state transitions, e.g., in the form of a finite state machine. This allows us to mirror the state of the production process in digital twins, including the logic that determines the transition to other production steps or states. Learning behavior on a component basis and forming a complete model from different components has the advantage that if one component is replaced, only the associated model must be re-trained without affecting the system-wide behavior model. This reduces the time it takes to adjust the complete behavior model if a component is exchanged. Another advantage of this approach is that component-based models are likely more reusable in other contexts, enabling collaborative learning and monetization of components.The digital human/context models have as main objective to achieve efficient human-centric intelligent control systems by learning and modelling human workflows, strategies and decisions as part of cooperative tasks. Environment and context parameters are integrated with the resulting operational models in order to address complex situations that are characterized by different levels of uncertainty. Two main types of model are developed: classification models that are able to recognize situations and models to predict human actions and decisions in a workflow . In this way, more intuitive user interfaces are developed, since the AI system knows how the human most likely will interact while keeping the human in control. This layer also contains services for business goal optimization, which are application specific. Potential applications of the platform are discussed in This layer provides innovative tools that will facilitate intuitive and efficient collaboration between humans, machines and AI systems allowing them to take advantage of each other’s strengths for more effective cooperative and intuitive task execution and decision making.The multichannel and context-aware interaction manager creates innovation by moving beyond traditional interaction mechanisms between humans and IT systems on the shop floor, such as command-line computer screen, pendant consoles and buttons. A good user interface is intuitive and does not require human operators to be educated with specific structures or behaviors. Voice commands could be such a solution, but due to the noise from machines voice is difficult to catch and process by AI systems on a shop floor. Instead, this component allows for multiple simultaneous input channels  that provide redundancy in the input, thus overcoming the decreased robustness of individual channels, e.g., due to variable lighting or noise on the production floor. To further improve the spontaneity of the human-machine interaction, context information is taken into account, such as the location of the operator or current production parameters .The intelligent decision support system (IDSS) will allow humans to consider extensive experience, expert knowledge, context information and empirical data to make rational decisions at the strategic or business level, e.g., to maximize the performance of the manufacturing system. In our proposed approach, traditional decision support systems (DSSs) which are widely used in the manufacturing domain will be extended with novel capabilities, e.g., digital twins or threat intelligence models. The integration of digital twins and IDSSs has great potential: the former do not have knowledge about enterprise business goals and constraints, while the latter require holistic and advanced simulation models to give recommendations. Thus, advanced data analysis techniques will make possible to rely on objective and evidence-based insights.The Federated Learning component aims at solving the problem of data collection for feeding or training AI models, while assuring the ownership and confidentiality of the data. In manufacturing, most  data and information are confidential because they relate directly to details of the production process, product characteristics, volumes, etc. By applying private set intersection (PSI) technologies , this coThe inter-manufacturing knowledge exchange serves as an interface for knowledge exchange across manufacturing sites or distinct manufacturing processes. Since there is a need to control what information is exposed and exchanged, rather than allowing open access to the local knowledge repository, this component contains a query engine to handle external requests. Such query engines also enable the realization of a federated query-processing mechanism over multiple sites.Our proposed platform also addresses security and authorization requirements on information and data sharing. Towards this end, the signcryption schemes component provides an effective and scalable solution, based on new cryptographic primitives, such as ciphertext policy attribute-based encryption (CP-ABE) schemes. The scheme will encrypt the data according to an access control policy based on a list of attributes and guarantees that the user’s keys are associated with the user’s descriptive attributes. Therefore, the data owner can freely define the access control policies that are used to encrypt the information, being assured that a user can decrypt the information if and only if their secret key matches the access policy used to encrypt the data. Thus, adopting this novel approach , this scThe presented framework incorporates AI in several components of an IoT-based manufacturing system. However, this introduces further concerns regarding the security and trustworthiness of these AI systems themselves.Artificial intelligence attacks, i.e., attacks on the AI algorithm, can take two forms: input attacks and poisoning attacks. The former consists in manipulating the input to the AI system during the operation phase so that it delivers the wrong results. Input attacks are relatively easy to launch and succeed since they do not require a manipulated AI system. Poisoning attacks, on the other hand, have to do with the corruption of the process used to build the AI model. In this case, inaccurate or mislabeled data are provided to the model during the training phase to manipulate the learning process. This type of attack can also be launched against federated learning; in this case, manipulated data or an algorithm of a member of the federation can result in the corruption of the global model.The defense against adversarial attacks, which is the core element of trustworthy AI, has recently received much attention . To protIn this section we discuss challenges regarding the implementation of the innovative AI-based components of the proposed system. Towards this end, relevant existing technologies and approaches are reviewed, limitations are identified, and new solution perspectives are presented.Digital twins are virtual, high-fidelity models of the current state and internal behavior of physical assets on the shop floor . These dhttp://far-edge.eu). The term edge computing covers various deployment options: from an on-premises datacenter to embedded computing, e.g., on AR glasses or on mobile robots. Techniques to make deep neural networks available for the latter form of edge computing start to find their way into production as more user-friendly tools become available. For instance, TensorFlow Lite allows converting a trained model for deployment on microcontrollers or embedded Graphics Processor Units (GPUs).Data-driven digital twins are built using machine learning. In this domain, deep learning (DL) is the dominant branch today, yet DL comes with its own problems: it requires domain knowledge to select the appropriate machine learning pipeline and it is very resource hungry in terms of computing and storage resources ,43,44. AFor the modelling of manufacturing assets, we propose a solution based on semi-automated construction of digital twins from components. This solution combines black-box data analytics  with grResearch on verbal-based human-AI communication has recently seen large advances. Modular dialog systems consist of, at least, (i) a natural language understanding (NLU) module; (ii) a dialogue management module; and (iii) a knowledge base. In order to extract information from human utterances and to track dialogue states, rules and machine- and deep-learning techniques have been used. So as to store knowledge, besides traditional databases, modern solutions make use of semantic web technologies to provide semantic processing of commands. As an alternative to modular systems, end-to-end approaches using deep-learning techniques are widely considered.The latest developments ,49 allowFor voice understanding, it is necessary to have a fair amount of data. Since the investigations in industrial settings are at an early stage, the data available for training the systems are scarce. To cope with such a problem, a promising solution is to develop a rule-based system as a first step to perform NLU and generate data for future training of supervised algorithms. In this approach, the system will automatically learn from new sequences while preserving previous knowledge. In order to store knowledge, semantic web technologies can be used ,51. ThesIntegrating traditional DSS with AI can increase the efficiency in the decision-making process and introduce higher levels of automation, resulting in the appearance of problem-oriented intelligent DSS (IDSS) or knowledge-driven DSS. Most of the ongoing efforts in this sense are still limited to the application of rule-based expert systems, or machine-learning or genetic algorithms  on raw dRegarding the realization of this component, we propose a hybrid approach, integrating knowledge bases  with data-driven methods. In particular, the use of advanced deep-learning models enable knowledge to be automatically synthesized from the raw multidimensional data collected from the entire manufacturing organization. The IDSS may also include continuous learning functionalities in order to increase its capabilities in time, extracting knowledge from the decisions taken by the human operators. This information can be used to provide better recommendations and to adapt behavior to changing conditions.Conventional security methods are not sufficient as industrial networks are foreseen to be frequently upgraded and their topologies are subject to changes. In this direction, adopting AI-based approaches is indispensable, in order to significantly enhance the network intelligence and security of industrial processes . TowardsBy exploiting advanced AI methods, the threat intelligence manager (TIM) component intends to model the dynamic interactions of Industry 4.0 subsystems and discover known and unknown attacks, while surpassing existing signature- and anomaly-based methods. This will provide an efficient solution for the detection, prediction and timely management and mitigation of cybersecurity incidents and abnormal traffic activity. The realization of this component can be based on the Beta mixture and hidden Markov models (BMM and HMM), aiming at accurately and effectively detecting and discriminating normal and attack data . In HMM,https://www.wikidata.org), which model general data, intra-organizational corporate/enterprise knowledge graphs model data relevant only to the organization itself, e.g., products or processes. It many practical scenarios, information is distributed across various sites, instead of being kept in a centralized location. This indicates the absence of a unique endpoint for data retrieval. In this context, federated query processing [The proposed system considers both the federation of knowledge using distributed querying, and of federated learning to avoid sharing raw data. In contrast to knowledge graphs such as Wikidata . Indeed, usually manufacturers do not want their data to leave their premises and be disclosed to cloud providers or third-party organizations. To this end, federated learning provides a solution enabling machine learning over distributed and decentralized datasets. Currently, federated learning is being adopted in different scenarios such as banking  and healA related approach is represented by private set intersection (PSI) that addresses the issue where different actors have information, often complementary, about some common business, but they cannot or do not want to share. To this end, PSI represents a solution that enables two parties holding a set of private information to identify the intersection of their datasets without revealing any other information except for the intersection itself. Some PSI protocol offers the possibility to perform simple operations on the intersection of the data sets . There aRegarding the query processing across sites, an efficient solution could be obtained by creating data summaries that are suitable for the underlying data representation and by extending the existing query engines to make use of such summaries for efficient federation.Several of the components of the proposed system have been partly tested in research and innovation projects during the last few years. This, in turn, illustrates their validity as separate parts and puts closer the solution described in this paper.In this context, an indicative example is the IoTEP, the Internet of Things Energy Platform . This soOne of the challenges to perform this in industry is the handling and labeling of data of very different types. In IoTEP, (transferable to this case) a built-in Representational State Transfer-based (RESTful) capability was created as aggregation functions. These features were included thanks to Comet (a short-time historic (STH) component of the FIWARE project ecosystem ) that peAnother example of the application of intelligent modules as the one suggested here, is in a solution for intelligent agriculture . In thisA platform based on the proposed architecture can be used for enhancing the performance of a variety of operations in the manufacturing domain. In the following we describe two indicative industrial application scenarios where different system components can be exploited for optimizing manufacturing logistics processes, and facilitating zero-defect manufacturing. For both scenarios we assume a realistic current practice and demonstrate what advancements the proposed system may bring to the relevant case.Many manufacturing companies are currently investing in autonomous mobile robots (AMRs) for supporting logistics operations regarding the transportation of goods from warehouse to working cells along the assembly lines or from the assembly line to dedicated areas for quality check purposes. Compared with automated guided vehicles (AGVs) that can only move along marked lines or wires in the floor, AMRs move flexibly in a factory, recognizing and avoiding obstacles at high safety levels and are easily reprogrammable. AMR-based deliveries are ideal for several types of factory because rigid automation like conveyors can be avoided, the system can work around people and machines at high levels of safety and reduce labor needs. In addition, accident risks are reduced by avoiding heavy vehicles like forklifts or tuggers in fast-moving intralogistics areas during peak activities. AMRs need to trade-off operational efficiency  with safety while achieving business objectives. Simultaneously, AMR downtime is undesirable and should be reduced as much as possible.This scenario assumes that a company has implemented a fleet of AMRs connected to a network infrastructure that allows for monitoring by company administrators for business and safety, and by the AMR supplier for maintenance. The current AMR production logistics is supposed to be carried out by the system illustrated in The manufacturing company is looking for predictable, deterministic material deliveries by AMRs that scale with the factory’s configuration and activities without tedious (re)programming. The deployed AMRs operate on highly dynamic production floor environments with people and vehicles. The AMRs are mostly individual units that plan isolated trajectories, frequently just stopping and waiting for an obstacle to pass. This results in missed delivery deadlines and stalling of downstream work cells. In addition, AMRs are currently unable to recover from system errors and failures internal or external to the AMR, for example, while positioning at a packing station or driving with a worn wheel. This results in expensive delays and downtimes. The complexity increases with every new AMR added to the fleet. Lastly, to obtain information about material delivery and AMR status, workers must use a PC or start an app on a tablet, which interferes with their activity.The AMRs need to be networked to the vendor- or enterprise-hosted AI cloud services to deliver real return on investments. Operationally, shop-floor supervisors and safety managers need situational awareness of the AMR fleet anywhere in the factory. AMR supplier companies may also want to offer remote maintenance services for their AMRs and thus need access to their customer networks. This opens opportunities for third-party attacks which can shut down production, or even compromise site safety. The solution must offer active threat-analysis that offers trustworthy insights that do not disrupt workflows through false positives or negatives. The fact of connecting the AMRs to the network infrastructure implies an inherent cybersecurity risk due to dynamic software vulnerabilities that affect the different software components and communication protocols. Indeed, the attack surface that could be exploited also implies safety risks for the human operators sharing space with the robots since an external attacker could gain control of them. Another particular risk arises since robots are equipped with many sensors and thus, they may leak confidential information about the shop floor. The manufacturing company needs the means to manage such risks. It also could cause big economic losses or reputation damage to the company if a serious accident is caused or if sensitive information is leaked.Advances leveraging the proposed platform: the platform can provide a secure AI infrastructure and associated services to optimize the AMR fleet business goals of delivery time guarantees and preventive maintenance by learning from patterns of business process models and anomaly detection techniques, as illustrated in Workers in this scenario can be equipped with a mixed reality (MR) application that presents live information about material delivery and that allows them to take corrective actions in case of problems. The AR/MR application can be available on wearable devices, being able to support gestures to reduce impacts on workers’ operativity. This component exploits features in the human-in-the-loop layer of the proposed system, concretely supporting interaction between people and AI.As unauthorized access to machines and data might compromise the physical integrity of human workers, communication between AMR, the platform and AMR supplier’s cloud back-end can be secured by signcryption schemes provided by the system. The manufacturing company can benefit from the integration of an instance of threat intelligence manager that will apply AI methods in order to identify anomalous data patterns that may be indicative of an ongoing cyber-attack. Using the private set intersection technologies in the TIM, the company will be able to assess which information is exchanged to the cloud-backend of suppliers, without requiring access to the internals of third-party services. Secure federated learning can be used to exchange information and module updates with the threat intelligence manager deployed on the site of another company participating in the federation through interfaces to exchange information about cyber threats and attacks.This scenario involves a company in the printing and offset manufacturing sector that aims to reduce costs associated with defects in the manufacturing process and enhance the security of its IT infrastructure. The company offers W2P services to its customers, allowing customers to design, edit and approve digital content as well as check the status of digital or offset printing process via a web interface. The printing process is quite complex and often requires manual interventions from the personnel. Defects along the manufacturing process have a major impact on the company’s financial losses. Despite the efforts spent on the formalization and standardization of the different procedures across the departments as well as on providing incentives to the personnel to provide products of high-quality, defective products  appear along the manufacturing process. Moreover, the acceptance of the final product is partially based on the customer’s subjective area.The process of manufacturing in the company includes several departments , running processes related to their specific tasks. In order to solve the defect manufacturing issue, the company stores and analyzes the deficiencies in the manufacturing process, thus holding a large dataset of production data  along with all the defects that have occurred , an approach illustrated in The analysis of these datasets has led to some first improvements in product quality, but there is still significant space for further improvements and minimization of financial losses and product quality improvement.The number and the diversity of characteristics of the orders do not allow for common process standardization.The data and information collected across the departments through manual and automatic procedures are solely processed by each department, without considering factory-wide optimization metrics.The deficiencies are observed, in the majority of the cases, during the quality control of the final product, thus leaving no space for corrective actions.The lack of personnel knowledge and training on the use of innovative tools that would automate the procedures , letting personnel only make the final decisions.The main reasons that create the problem and still remain unsolved are:At the same time, the company aims to reduce the cyber-attack surface of its IT infrastructure that is distributed in two sites. To create their printing file or obtain real-time information about the file printing status, clients connect to the W2P web service in the first site. Also, customer relationship management (CRM) is installed on this site. The servers in this site are connected to the management and information system (MIS) of the printing machines in the second site, in order to schedule the timing of the printing job according to the rest of the printing jobs in the factory. A high-level architecture is depicted in The connection to the second site of the company where the production line, warehouse and financial services are hosted, is achieved through two hardware firewalls with basic security services subscriptions enabled. On the second site, the portal server is used to store the uploaded files which will be used to the production line of the company, while the management and information system (MIS server) provides financial information, status of the printing process, and other types of information for the printing job status, such as warehouse information.Since customers have access to a web server in the company network, and because the IT systems are physically distributed over two sites, there are many possible points of attack. Thus, W2P service offers an attack surface to cyber attackers to intrude the network and gain access to customers’ financial data and even control the printing machines, leading to the disruption of the company production line and even to potential safety risks for the human operators.Advances leveraging the proposed platform: several elements of the proposed system can impact the business of such a company, as illustrated in Components of the proposed system can also be used in a principled approach to mitigate the security and data leak risks, as it can be seen in In testing and validation procedure, the goal is to define a set of KPIs per potential use case and validate the performance of our proposed AI-based approach via execution of the trials and validation. The evaluation plan includes technical, business and ethical validation metrics, since not only the technical requirements will be assessed but also the business feasibility and impact as well as the ethical issues related to the introduction of novel AI concepts and models.The first and primary business impact of the proposed approach is on the manufacturing domain. Namely, through the proposed tools and components, manufacturers will be capable of realizing agile production processes and improve the quality of products and processes. This will help them to be more competitive in the market and thus increase their market share. Apart from this main business impact, additional business impacts can be identified as discussed below.The federated intelligence layer introduced in our approach enables new business models , as well as the collaboration of different industries towards the creation of digital twins. Any digital twin of a physical object captures and exhibits the behavior of the physical object to a certain accuracy and with respect to specific functionalities which may be of interest to different people inside or across different manufacturing plants. For digital twin value to be recognized , the insThe presented approach boosts the innovation potential of IT-, IoT- and AI-related companies through the definition of an architecture that facilitates the collaboration between manufacturing machinery, AI and humans with the already explained anticipated benefits. IT companies can offer open interfaces or adopt the presented interfaces so that their solutions are extendable to support novel collaborative AI applications. Last but not least, companies with expertise in AI for manufacturing can create significantly higher revenues by being capable of integrating their components with IoT and IT systems from different vendors/creators.A novel architectural approach has been proposed, that extends the RAMI 4.0 architecture in the context of adopting AI systems in the manufacturing area. At the core of this approach is the provision of components for timely data collection, processing and curation, relying on the dynamic instantiation of data pipelines, while addressing security, privacy and confidentiality concerns across the physical and virtual entities. The collected data and information models are transformed into AI-enabled functional intelligence, leading to business knowledge, actionable insights and informed decisions, while being capable of recognizing complex events and process deviations that cannot be captured easily and in a timely way through human judgment. The provided functional intelligence is leveraged by components for secure communications, federation and exchange of data, knowledge, trained AI models or optimized AI algorithms across manufacturing sites and stakeholders, as well as by components that support efficient interactions between humans and AI systems through intuitive interfaces and facilitate informed and timely decisions regarding the operation of the manufacturing system.The study of industrial applications regarding the optimization of the manufacturing logistics processes, and the facilitation of zero-defect manufacturing, illustrated the potential for practical applicability of the proposed architectural elements. Finally, the elaboration on the challenges regarding the implementation of the key AI-based components of the proposed system and the review of existing solutions identified areas where new approaches are needed to overcome current limitations and deliver effective and scalable solutions that can be exploited in new business models.


Analytical modeling of soft pneumatic actuators constitutes a powerful tool for the systematic design and characterization of these key components of soft robotics. Here, we maximize the quasi-static bending angle of a soft pneumatic actuator by optimizing its cross-section for a fixed positive pressure inside it. We begin by formulating a general theoretical framework for the analytical calculation of the bending angle of pneumatic actuators with arbitrary cross-sections, which is then applied to an actuator made of a circular polymer tube and an asymmetric patch in the shape of a hollow-cylinder sector on its outer surface. It is shown that the maximal bending angle of this actuator can be achieved using a wide range of patches with different optimal dimensions and approximately the same cross-sectional area, which decreases with pressure. We also calculate the optimal dimensions of thin and small patches in thin pneumatic actuators. Our analytical results lead to clear design guidelines, which may prove useful for engineering and optimization of the key components of soft robotics with superior features. Recent decades have seen explosive growth in the robotics industry ,2,3. TheThe rise of the 3D printer and drops in the cost of traditional manufacturing methods have led to significant expansion in the soft-robotics industry ,18,19,20However, soft robotics in its current evolution faces a number of challenges ,24,25. FScaling down conventional rigid actuators comes with three major issues: (i) as a design is scaled down, the cost of production and manufacture rises rapidly; (ii) the high rigidity of actuating mechanisms is not ideal in many environments; and (iii) delivering substantial force from smaller mechanisms is a significant engineering challenge.The third issue has been partially solved with the advent of small-scale robots ,31,32,33In summary, soft robots today are too expensive and lack a general fabrication platform, however, recent advances in drawing soft polymer fibres may provide solutions to these problems. It has been shown that soft polymers such as polyurethane can be thermally drawn down to extremely narrow tubes (fibers) at the scale of a few hundred microns ,40,41. MThis low cost of production and the ability for significant size variation can address the issues previously discussed, making soft polymer tubes potentially superior candidates to existing soft robotics technologies. Using such materials as the sole constituent of a soft robotic actuator requires the creation of new pneumatic designs, since existing models like bi-bellows are only possible with the design flexibility offered by the relative precision of 3D printers. A brief review of existing technologies makes it evident that asymmetry in design is required to achieve pneumatic actuation.In this paper, we present a systematic theoretical study to optimize the cross-section of soft polymer tubes which may serve as a new material base for soft pneumatic actuators. We specifically maximize the bending angle of a soft pneumatic actuator realized as a circular polymer tube with an asymmetric patch in the shape of a hollow-cylinder sector attached to its outer surface. The general expression for the actuator’s bending angle is analyzed to calculate the optimal parameters of the patch and formulate useful design guidelines that can facilitate engineering and optimisation of the key components of soft robotics.E, length L, and cross-sectional area z axis and have a plane of symmetry In the design of pneumatic actuators, engineers are usually interested not only in maximizing the forces produced in response to a given pressure but also in maximizing the deflections . To calcP and the cross-sectional area A of the pressurized volume of the tube, e between the neutral axis and the centre of pressure. In the adopted geometry this offset is equal to the y coordinate of the neutral axis and given by . Hence,E of the polymer. Fourth, if the thicknesses of the patch and the main hollow tube of the actuator are alike, the optimal angular width of the patch varies between 149° at low pressures, Summarizing the above results, we can formulate the following general conclusions and design guidelines for soft pneumatic actuators made of hollow polymer tubes. First, the bending angle of soft pneumatic actuators scales linearly with the ratio of their length to the inner tube radius and can be maximized for a fixed pressure by tuning the cross-section of the asymmetric patch. Second, there is a tradeoff between the width and thickness of the optimal asymmetric patch allowing one to achieve almost the maximal bending angle using a wide range of patches. Third, the bending angle of thin pneumatic actuators (with We have optimized the cross-section of a soft pneumatic actuator to achieve its maximal bending for a fixed actuation pressure. By applying a general analytical model of pneumatic actuators with arbitrary cross-sections to an actuator realized as a circular polymer tube with an asymmetric patch in the shape of a hollow-cylinder sector, it was shown that the strongest deflection of the actuator is achievable with different optimal dimensions of the patch, which depend on the applied pressure. We also calculated the optimal patch dimensions for thin actuators and formulated general design guidelines for soft pneumatic actuators in terms of their relative material and geometric parameters. We believe that our results will benefit the development of soft robotics and facilitate the design of new integral components of soft robotic systems.


Wearable sensor technology already has a great impact on the endurance running community. Smartwatches and heart rate monitors are heavily used to evaluate runners’ performance and monitor their training progress. Additionally, foot-mounted inertial measurement units (IMUs) have drawn the attention of sport scientists due to the possibility to monitor biomechanically relevant spatio-temporal parameters outside the lab in real-world environments. Researchers developed and investigated algorithms to extract various features using IMU data of different sensor positions on the foot. In this work, we evaluate whether the sensor position of IMUs mounted to running shoes has an impact on the accuracy of different spatio-temporal parameters. We compare both the raw data of the IMUs at different sensor positions as well as the accuracy of six endurance running-related parameters. We contribute a study with 29 subjects wearing running shoes equipped with four IMUs on both the left and the right shoes and a motion capture system as ground truth. The results show that the IMUs measure different raw data depending on their position on the foot and that the accuracy of the spatio-temporal parameters depends on the sensor position. We recommend to integrate IMU sensors in a cavity in the sole of a running shoe under the foot’s arch, because the raw data of this sensor position is best suitable for the reconstruction of the foot trajectory during a stride. Wearables have become increasingly important in many fields of our everyday life. Among applications in medicine, workplaces, and many others, the sports domain was one of the early adopters of wearable technology. The reasons for the quick spread of small body-worn sensors in sports was due to the manifold advantages of the technology for athletes, researchers, and the sports industry. Using wearables, athletes can utilize low-cost sensor technologies in order to enhance their performance, prevent injuries, and improve their motivation . Sports One sport where wearable technology already has a great impact is endurance running. Both professional and recreational runners track themselves and use online platforms like Runtastic  or Strava  to monitor their training progress and performance. Three heavily used sensor technologies in endurance running are Global Positioning System (GPS) trackers, heart rate monitors, and inertial measurement units (IMUs). While GPS trackers like smartwatches or smartphones are utilized for visualizing the running track and providing real-time feedback on pace and distance , heart rIn this context, IMUs can be used in various ways. Researchers developed body sensor networks with multiple sensors in order to reconstruct the movement of different extremities in a synchronized manner . The advHowever, publications using foot-mounted IMUs differ not only in the computed spatio-temporal parameters, but also in the position of the IMU sensors on the running shoes. Shiang et al. , FalbriaTo our knowledge, the effect of the IMU sensor position on the foot with respect to raw data quality and accuracy of spatio-temporal parameters has not been evaluated yet. Peruzzi et al.  evaluateWe contribute an in-depth analysis by comparing different sensor positions with respect to the raw signals as well as their suitability for the computation of spatio-temporal parameters. We base our evaluation on a study with 29 subjects, which wore running shoes with IMUs attached to the instep, the heel, the lateral side of the foot, and within the cavity of the running shoe. We compare the similarity of IMU raw data at different sensor positions by computing Pearson’s correlation coefficients between the individual sensor positions’ raw data. Besides, we compute and evaluate temporal stride features based on state-of-the-art event detection algorithms and spatial features using a zero-velocity-based strapdown integration algorithm.The temporal parameters we evaluated were stride time time tgc . One strThe spatial parameters based on the translation of the foot were stride length We used the sole angle and the range of motion as spatial parameters based on the orientation to evaluate the sensor positions . The solWe collected data of 29 amateur runners  with a mean age of e instep .n measured with those sensors will be denoted asx, y, and z denote the vector components along the axis of the respective sensor ((1)Vector pair superior/inferior direction: The subjects were asked to stand still with both feet on the ground. Thus, the accelerometer of all sensors measured the gravitational acceleration in the sensor frame. The (2)Vector pair medial/lateral direction: The subjects rotated their feet on a balance board, which only allowed for a rotation in the shoe frame’s sagittal plane. A gyroscope in the shoe frame measures the angular rate of the rotation on the medial/lateral axis. The medial/lateral axis of the shoe frame corresponds to the principle component of the angular rate data during rotation in the sensor frame. The For the study, we used miPod IMU sensors . The acce sensor . The senr et al. , we set s et al. . Additio routine  was perfx, y, and z as the axis in the shoe coordinate system from now on.Using these vector pairs, we computed a subject dependent rotation matrix for each sensor location, which rotated the IMU-data in the sensor frame into the shoe frame using an adapted version of the Whaba algorithm ,25. AfteThe motion capture ground truth system consisted of 16 infrared cameras and sampled the positional data of the retroreflective markers with a sampling rate of l et al. . For ourl et al. , the refThe sensors and the motion capture system were synchronized using an adapted version of the wireless trigger introduced by Kugler et al. . Due to In the described set-up, each subject was asked to run 50 times through the motion capture volume. We controlled for speed by capturing different number of trials in different velocity ranges of 2–6 m/s . Using tAn example stride for the four IMU sensors, which was segmented from IC to IC, is depicted in We used a combination of different existing stride event detection algorithms to find IC, MS, and TO in the IMU signal. IC and TO were computed by finding maxima in the angular rate data of the sagittal plane. According to Falbriard et al. , these ti-th ground contact at the local minimum of the gyroscope in the sagittal plane nMS,ii-th TO event For the detection of TO, we used the first maximum in the angular rate data after the maximum related to IC in a time range of 400 ms . We addeFor the computation of the spatial parameters we implemented a strapdown integration algorithm to reconstruct the trajectory of the foot using a zero-velocity update to reduce IMU drift. The trajectory algorithm is an adapted version of the sensor fusion algorithm introduced by Rampp et al. . It consThe orientation estimation is based on the zero-velocity assumption. During MS the foot is assumed to be flat on the ground with a velocity of 0 m/s. Therefore, we can reset the orientation and the velocity for each stride during MS. For running, this assumption is often violated due to the dynamic nature of running for high running speeds. Besides, in the case of forefoot running no flat-foot phase exists. However, we could show in one of our prior publications  that thei-th MS index The following computational steps were applied to each running stride. Using the zero-velocity assumption, we initialized the orientation After the initialization, we computed the orientation of the sensor during one stride using a quaternion-based forward integration. For this work, we use a vector representation for quaternions:The quaternion formulas ,34.(5)qIn Equation , the quaThe quaternion sequence was used in two ways. First, the orientation of the sensor was computed in Euler angle representation by converting the quaternions to roll, pitch, and yaw:n to the initial position at MS. In the global frame, the magnitude and direction of gravity was known, and thus we could remove it from the movement acceleration:Second, the quaternion sequence was used for the gravity removal of the acceleration signal. The accelerometer constantly measures not only the acceleration of the foot movement, but also the gravitational acceleration. Using the determined quaternion sequence, the acceleration data As a last step, the translation of the sensor was computed by a dedrifted double integration of the gravity corrected acceleration signal The individual steps of the trajectory computation are visualized for one samples stride in Based on the segmentation indices, the parameters stride time Stride length x and y component of the translation due to the fact that the running path through the ground truth system was also flat. Thus, an error of the translation in z-direction was neglected.Please note that we assumed level running and thus only use the i-th MS at i-th ground contact, the orientation in the sagittal between the i-th IC i-th ground contact nor for the In order to compute the angle parameters, one integration step was missing. We defined our stride from the We used Equation  for the Equation . FinallyEquation  and concIn this orientation sequence, the sole angle is the angle obtained at the first IC C and TO .To evaluate the effect of the sensor positions, we compared the raw IMU signals of the individual sensor positions with each other as well as the errors for the IMU-based spatio-temporal parameters.For the comparison of the raw signals, we used Pearson’s correlation coefficients . We compTo evaluate the spatio-temporal parameters with respect to the sensor positions, we computed the error of individual parameters for all sensor positions. For this work, we defined the error In this formula, In order to understand the impact of running speed on the spatial parameters, we evaluated stride length The idea of the error measure Edstride a as wellEa→[nms] b for theEa→[nms] . We see Overall, no sensor position clearly outperforms all the other sensor positions even though the cavity sensor provides the best results for the translational parameters based on the reconstructed foot trajectory. In the following paragraphs, we will discuss the raw data comparison, the temporal parameters as well as the spatial parameters individually.The correlation coefficients are higher for the gyroscope signals than for the acceleration signals . The reaAdditionally, the correlation coefficients during swing phase are higher than during ground contact phase . In partThe correlation coefficients for the accelerations of the full strides show that the correlations to the instep sensors always yields the lowest values. We argue that the instep sensor is exposed to the highest motion by the deformation of the foot. During ground contact the arch of the foot and the forefoot flatten, which causes the upper of the running shoe to deform. This deformation of the upper also causes the laces and the tongue of the shoe to move and, consequently, the instep sensor as well.More detailed analysis of the correlations reveals that the closer two sensors are located to each other, the higher their pairwise correlation. This makes sense, as we already discussed that the accelerations for the sensor positions differ depending on the distance to the joints causing the rotations. Thus, spatially close sensors have similar distances to those centers of rotation and and might be similarly effected by deformations of the shoe.One limitation of our approach to comparing the raw signals is that the Pearson’s correlation coefficients might be influenced by differences in the detected events. Depending on the sensor positions, the events might be detected with a slight time shift. However, the high correlation coefficients during swing phase show, that the signals temporally match well and are not heavily influenced by time shifts.In summary, the correlation coefficients show that the signals of the sensors at the four positions of the running shoe vary which causes different results for the spatio-temporal parameters. Especially the larger differences during ground contact have an effect on the accuracy of the parameters.The results for stride time d et al.  who repoGenerally, both the median errors and the IQRs for ground contact time al plane , the dynal plane . Moreoveal plane  showed tFor the orientation parameters, the cavity sensor positions shows the smallest median errors for both the sole angle and the range of motion. While the differences in accuracy between the sensor positions is smaller for the median error of the range of motion (2.3 results .If no bias correction is applied during the event detection of IC the median errors are better for the heel and the lateral sensor position. Falbriard et al.  explaineFor the stride length parameter the cavity position performs best, followed by the lateral position. The heel and the instep sensor position perform worst. These observations also hold for the different speed ranges a. We canThese movements and consequently the errors during MS harm the validity of the zero-velocity assumption which is the basis for the strapdown integration algorithm. Due to the fact that the sensor positions seem to be affected by additional movements or noise sources during MS to a different extent, the performance of the zero-velocity-based strapdown integration algorithm for the trajectory computation is different for the evaluated sensor positions.The errors of the average stride velocity for the different sensor positions correlate with the ones for stride length. The reason for this correlation is that the average stride velocity is computed by dividing stride length by stride time. As the stride time computation works well for all sensor positions, the main error source for average stride velocity is the computed stride length. However, we can see from the small median errors and IQRs that average stride velocity can accurately be computed with foot worn IMUs.Our results indicate that the raw signals and thus the results for the different spatio-temporal parameters differ for the evaluated sensor positions. This implicates that when comparing studies of IMU-based spatio-temporal parameters with each other, the sensor position should be considered as a source of difference in future review studies. Besides, adaptions to the algorithms like changing the detection algorithm of IC for the computation of the sole angle can result in performance boosts individual sensor positions. Thus, porting algorithms to other sensor positions can be possible, but algorithms might have to be adapted.Nevertheless, we recommend the cavity sensor position due to the unobtrusive and firm attachment inside the running shoe sole. It is least affected by any additional movement than the actual foot movement and thus has the highest raw signal quality. However, we want to note at this point that this evaluation is purely based on the signal quality and the accuracy of the resulting spatio-temporal parameters and that we did not consider usability aspects. While the unobtrusiveness of the cavity sensor position is good for a smart shoe application with fully integrated sensors, it might not be a good solution for a sensor system which should be usable with different pairs of shoes. In this scenario, all the shoes would need a cavity and the sensors would need to be put under the sockliner before each run.We presented an evaluation of the effects of four different IMU sensor positions on the accuracy of IMU-based endurance running parameters. We conducted a study with 29 subjects which were equipped with four IMUs placed inside a cavity in the sole of the running shoe, on the heel, the lateral side, and the instep of a running shoe. We compared the raw data of the individual sensor positions and implemented algorithms for stride segmentation and the computation of the trajectory of the foot, respectively, the running shoe. Using the data acquired during the study, we could show that the raw signals of the IMUs differ for the sensor positions, especially for the acceleration during the ground contact phase. We showed that all the sensor positions could accurately measure stride time, but not ground contact time due to the large speed dependency of the fiducial point we used to detect toe off. The angle parameters—range of motion and sole angle—were hard to compare for the different sensor positions, as they were affected by the accuracy of the IC event detection algorithm. Finally, we showed that the cavity sensor outperforms the other sensor positions for the computation of stride length, because this sensor positions seems not to be exposed to movement artifacts of the upper of the shoe.Thus, we can conclude that the sensor position has an effect on the accuracy of different IMU-based running parameters due to the differences in the acquired raw signals. From a data processing perspective, we recommend to use IMU sensors inside the cavity of a running shoe even though it only outperforms the other sensor positions for the reconstruction of the trajectory using a zero-velocity-based strapdown integration algorithm.In the future, the event detection algorithm for the different sensor positions should be investigated further. Because a sample-to-sample synchronization was not possible in our study set-up, we could not evaluate the accuracy of the actual IC/TO events but only the parameters deduced from those events. Our results indicate that, both for the IC and TO event, the biases from the maxima in the pitch angular velocity to the actual events are different for the different sensor positions. Improving the detection of the IC and TO event would not only increase the accuracy of ground contact time but also the stride angle parameters. Further, the larger errors for stride length with higher speeds should be further investigated. Especially for professional athletes, who run with speeds higher than 5 m/s, the errors in stride length are very high.