Collaborative part transportation is an interesting application as many industrial sectors require moving large parts among different areas of the workshops, using a large amount of the workforce on this tasks. Even so, the implementation of such kinds of robotic solutions raises technical challenges like force-based control or robot-to-human feedback. This paper presents a path-driven mobile co-manipulation architecture, proposing an algorithm that deals with all the steps of collaborative part transportation. Starting from the generation of force-based twist commands, continuing with the path management for the definition of safe and collaborative areas, and finishing with the feedback provided to the system users, the proposed approach allows creating collaborative lanes for the conveyance of large components. The implemented solution and performed tests show the suitability of the proposed architecture, allowing the creation of a functional robotic system able to assist operators transporting large parts on workshops. The emergence of collaborative robotics is changing the way of developing new robotic applications, especially in those cases involving cooperative tasks between humans and robots. This kind of cooperative task raises many technical challenges, ranging from the use of force feedback to guide the collaborative process  to the aRobotic part transportation is a compelling application as many industrial sectors such as the aeronautic ,4 or autThis paper presents a path-driven dual-arm co-manipulation architecture for large part transportation. This architecture addresses three key aspects of the collaborative part transportation task: (1) Human-driven mobile co-manipulation, (2) soft superposition of navigation trajectories to the co-manipulation task to ensure safety zones within the workshop, and (3) robot-to-human feedback to guide and facilitate the collaborative task. The architecture tackles these three topics, proposing a new scheme that addresses issues for the industrial implementation of this kind of systems like safety and real-time feedback of the process state. The implementation and evaluation of the architecture demonstrate its suitability for cooperative applications in industrial environments.The paper is organized as follows. Human-robot manipulation is a broad research topic, with multiple approaches proposed for a wide range of scenarios and applications. From classical scenarios with industrial robotic manipulators , to the Force control is one of the most studied fields in human-robot collaboration, with a wide range of algorithms and approaches to offer a seamless interaction based on force. Peternel et al.  propose Additionally, the use of guides and predefined trajectories for dual-arm and co-manipulation tasks appear in different research activities. Gan et al.  present Multi-robot systems are an alternative for force-based large part co-manipulation. Hichri et al. address the flexible co-manipulation and transportation task with a mobile multi-robot system, tackling issues like the mechanical design  and optiThe use of Artificial Intelligence for mobile manipulation and cooperative tasks is also a recurrent research topic, as it adds mechanisms to tune and optimize control parameters. Zhou et al.  propose From the user point of view, it is also important to include human and social factors in the design of collaborative applications as the lack of transparency in robot behavior can worsen the user experience, as shown by Sanders et al. and Wortham and Theodorou ,24. TherThe main contribution of the presented work is the usage of navigation paths in force-based mobile co-manipulation. This addition allows modifying the commands generated from the force information and providing mechanisms to limit the robot’s workspace. This feature enables the definition of safety areas within the workshops, a critical issue in the implementation of this kind of application in a production environment.Collaborative part transportation is an appealing application, as many industrial sectors such as the aeronautic or automotive one require moving large parts among different areas of the workshops, spending a large quantity of workforce on tasks with no added value. Therefore, it is interesting to offer robotic solutions that allow collaborative transportation, paying attention to different aspects such as the control algorithm or robot-to-human feedback.During large part transportation by humans, both actors agree (implicit or explicitly) on an approximate path to follow during the process. Additionally, humans can indicate their destination during the task. Therefore, the robot should include some feedback indicating which direction they are (or should be) moving.As robots act as assistants of the human, robots will only advance in the path when the operator moves the part in the defined direction. Therefore, humans always have the master role in the co-manipulation task.Industrial workshops generally include lanes used for the transit of vehicles and large machines. The mobile co-manipulation system should offer mechanisms to limit the co-manipulation areas and restrict them to the allowed zones when required. These safety measures are mandatory for the industrial implementation of this kind of solution.In an initial step of the development, the large part transportation process has been analyzed to extract the key elements of the task, as well as the requirements to be transferred to the robotic solution:Force Management Layer: This initial layer is in charge of generating twist commands for the mobile robotic platform based on the force information received from the arms. This force information could be provided by force/torque sensors attached to the grippers or directly by robots equipped with internal sensors.Moving Average Filter Layer: This second layer filters the received twist commands to smooth the velocity.Path Management Layer: This third layer modifies, if necessary, the smoothed twist commands to ensure that the robotic platform is within the safety lanes defined in the workshop.User Interface Management Layer: This last layer is in charge of presenting the co-manipulation feedback to operators, using different cues to this end.To cope with the previously listed requirements, a four-layer architecture is proposed:This four-layer architecture allows the mobile co-manipulation task, including different modules for the robot control besides the generation of robot-to-human feedback. It ensures that the system controls all the process steps, from low-level control to high-level interaction feedback. The following sections provide further information about the different layers and their specific features.G is the gravity vector The first step is to estimate the external forces applied on the left arm G is the gravity vector The external torques applied on the left arm The overall external force V that will be sent to the mobile platformX and Y, respectively, while Z.These force and torque vectors V, the idea is to define a force and torque range where the robot will move. If the sensed forces and torques are below a predefined threshold, the twist values will be equal to zero. These twist values will increase linearly as forces and torques increase, defining a maximum allowed force and torque, as well as a maximum robot speed. It will allow defining different force/torque and velocity profiles, profiles that can be used on different phases of the part transportation such as the fast movement between stations of the workshop or the precise and slow positioning of the robotic platform at the part loading area.In the generation of this twist vector X and Y, The linear velocities Z, The angular velocity Moving Average Filte Layer.This twist vector moving average filter [K defines the size of the filter and K twist vectors.This layer applies a classical e filter  to the tPath Management Layer, layer that will modify it in order to fit in the path provided to the algorithm.This filtered vector The main idea of this layer is to define a corridor where the robotic platform can move freely, limiting the movements when the robot is trying to go outside the borders of the lane. It allows the creation of some virtual walls at both sides of the lane, as shown in P and the current pose of the robotic platform in the map M poses asTherefore, the filtered twist vector L is calculated, a coordinate system with the origin in X pointing to pose L is calculated using the following equations:A defines the unitary vector pointing from path pose i to pose C defines the unitary vector pointing upwards in the map andB defines the unitary vector pointing perpendicularly to the lane border.In an initial step, a new coordinate system Hpathi+1 . The calA, B, and C define the rotational part of the transformation matrix and vector Based on these three unitary vectors, the new coordinate system is calculated asL is defined, the poses of the robotic platform Once this path frame X and Y, and At this point, the theoretical next robotic platform pose is calculated asP and the maximum distance d from the path. Specifically, the corrected next platform pose is calculated asX and Y.In the next step, the corrected next platform pose Z as it will not be limited nor modified by the Path Management Layer.Finally, the corrected twist values P and the maximum distance d. Additionally, this vector User Interface Management Layer to generate the appropriate feedback for the co-manipulation process.This new twist vector Movement direction: The movement direction is directly generated using twist vector Direction to the next path pose: The second cue defined to guide the co-manipulation process is the direction vector to the next path pose This last layer will manage the generated twist commands, besides the current robot status, to provide helpful feedback to operators. Specifically, two different feedback cues have been considered:Additionally, some audio signals have been added to indicate when the path-driven mobile co-manipulation starts and when the destination has been reached. These audio cues would complete the feedback sent from the robot to the user to guide the mobile co-manipulation task.The proposed architecture has been implemented in a dual-arm mobile platform designed for the manipulation of large parts in industrial environments, see movement direction generated in the User Interface Management Layer. Therefore, the robot’s front and rear lights blink when the robot moves forward or backward, while the sidelights blink when the robot moves in these directions.An omnidirectional mobile platform equipped with Mecanum wheels . The plaKuka LBR iiwa robots with a payload of 7 kg, collaborative arms equipped with torque sensors in each joint. These robotic arms can provide force and torque information estimated from the torque sensors placed in each joint. Therefore, this implementation does not require an external sensor to acquire force and torque information.Two The arms are also equipped with automatic tool exchangers, a suction system, and vacuum cups to grasp different types of large objects and parts.An additional IO module is also available to manage the tool exchangers and the suction of the vacuum cups.direction to the next path pose on the ground in the front part of the robot as shown in A small size projector has been installed in the front part of the robot. This projector is used to display the From the software point of view, a PC placed inside the robotic platform executes the four modules of the architecture. All these modules have been implemented in C++ as ROS nodes. An additional HTML5 server provides the web interface that shows the arrows and cues used to guide the co-manipulation process.Finally, an HMTL5 based general User Interface (UI) has been developed. This interface allows commanding the system, triggering and canceling the execution of the mobile co-manipulation at any moment. A tablet is used to display this UI, enabling operators to utilize this interface at any place within the workshop.shelf and inspection station, 10 m away from each other  were used, 0.250 m and 0.500 m, defining lanes with a width of 0.500 m and 1.0 m, respectively. It will serve to assess the impact of the lane width on the trajectory deviation.During the transportation, two different maximum distances from the path  with two different lane widths. Therefore, each of the five subjects performed four different transportation processes, moving the part in both directions . Therefore, users tend to use all the available space to guide the robot; it is especially notorious when users reach bends along the route. Additionally, lower mean deviation values have been observed with a maximum allowed distance of 0.250 m. However, the difference with the maximum distance 0.500 m is not that large (only 0.050 m more) based on the difference in the lane width. Finally, users can cover the path in less time with a wider lane, as they can guide the robot easily and even take shortcuts in sharp bends due to the width of the path.d. Additionally, it did not impact the mean deviation value greatly as the projected arrows guide the users and allow them to re-enter the path after sharp bends or unexpected obstacles.As a general conclusion, the experiment subjects were able to carry out the part transportation in less time and more comfortably with a greater maximum distance This paper presents a novel architecture for mobile co-manipulation. The architecture includes the capability to add a path to guide the part transportation process and ensure safety by creating virtual lanes in the robot’s workspace. Additionally, the architecture incorporates a specific module to generate robot-to-human feedback to assist the process and improve the user experience in the collaborative task. Therefore, the proposed work covers all the different steps of the mobile co-manipulation for large part transportation, starting from the force-based control algorithm and finishing with the generation of suitable feedback for the system users.The proposed approach has been implemented and tested in an industrial workshop environment, assisting operators in the transportation of large carbon fiber parts within different workstations. The obtained results show the suitability of the architecture, highlighting the utility of the robot-to-human feedback generation to guide operators along the defined safe paths of the workshop. Moreover, the addition of projected arrows helps operators to keep close to the nominal path even with wide co-manipulation lanes.As future steps, several research paths have been identified. On the one hand, it would be interesting to investigate different cues and communication methods to improve co-manipulation tasks to create a seamless interaction between humans and robots. On the other hand, many parameters of the algorithm must be tuned manually by experts. Therefore, it would be interesting to include Artificial Intelligence techniques to extract these data from the information gathered during the co-manipulation processes as currently all this data is not further used nor exploited.


At present, deep learning-based medical image diagnosis had achieved high performance in several diseases. However, the black-box nature of the convolutional neural network (CNN) limits their role in diagnosis. In this study, a novel interpretable diagnosis pipeline using the CNN model was proposed. Furthermore, a sizeable melanoma database that contains 841 digital whole-slide images (WSIs) was built to train and evaluate the model. The model achieved strong melanoma classification ability . Moreover, the proposed model outperformed the existing schemes in terms of accuracy that is 20 pathologists (0.933 vs 0.732 accuracy). Finally, the gradient-weighted class activation mapping (Grad-CAM) method was used to show the inner logic of the proposed model and its feasibility to improve diagnosis process in healthcare. The mechanism of feature heat maps which is visualized through a saliency mapping has demonstrated that features learned or extracted by the proposed model are compatible with the accepted pathological features. Conclusively, the proposed model provides a rapid and accurate diagnosis by locating the distinctive features of melanoma to build doctors' trust in the CNNs' diagnosis results. Malignant melanoma is a melanoma cell carcinoma , 2, and In clinical routine or practice, high accuracy for the detection of malignant melanoma is of utmost importance to make these systems trustworthy for doctors and practitioners in the hospital system. For this purpose, various histopathology features have been associated with the diagnoses of melanoma disease in numerous patients , and sevIn addition to the discrimination power, model interpretability is another crucial issue for neural networks, especially in life-saving medicine and development of an intelligent healthcare diagnostic system for the hospitals –20. In lThe remaining paper is organized as follows. In The proposed diagnosis pipeline consists of two parts, as illustrated in As shown in the first row, the model was trained in a patch set sampled from WSIs. Furthermore, the WSI diagnosis was generated by counting the CNN inference. The second row shows that Grad-CAM has generated the heat map of critical patches after model prediction.The training and validation of previous studies have been limited by the small amount of data, which portend a risk of selection bias. Furthermore, these studies have not been focused on early prediction of the malignant melanoma and to make these systems trustworthy for both doctors and patients. In proposed system, we collected 841 H&E stained whole-slide histopathology images for the present study and built a pathological image database from March 2018 to May 2019. This dataset is generated by collaboration with the Central South University Xiangya Hospital (CSUXH). In this dataset, we have stored three hundred and ninety-two (392) melanoma WSI symtoms and four hundred and fourty-nine (449) nevi WSIs which were collected during the aforementioned time interval. In order to verify labels of the collected WSI (both melanoma and nevi), we have consulted five responsible board-certified pathologists preferably those residing in closed proximity to streamline the proposed work methodology verification.∗256 patches using the no overlapping cutting method. Furthermore, we have filtered the blank patches through the OTSU method which is computed using the following equation:ωo and ω1 represent the expected probabilities of the two classes which are separated by a threshold value t. Furthermore, metrics σo2 and σ12 are used to represent variances of the concerned classes. The patches of WSI as described above are shown in ∗256-pixel patches which are used to train the proposed model in real environment of the hospital systems. The training dataset, validation dataset, and test dataset were divided in a ratio of 7 : 1, 5 : 1, and 5 : 1. Additionally, patches from the same patient data can only be divided into one dataset to ensure that data is not cross-contaminated and is not manipulated.Model training is one of the challenging tasks in deep learning-enabled models particularly for accurate and precise detection of various diseases, i.e., malignant melanoma in this case. In order to train the proposed CNN-based prediction model, we have built a dataset by sampling lesion patches from WSIs which are collected by the Central South University Xiangya Hospital (CSUXH) during the aforementioned time interval. Additionally, pathologists are consulted to mark the lesion area in the collected images which is quite useful in the development of a proper prediction system. Due to the enormous (comparatively large) size of WSIs , these WSIs are potential candidates for the CNN-enabled prediction system after being divided or cut into valuable patches as shown in In this dataset, melanoma and nevus patches are shown separately where MM and NV are used to represent melanoma and nevus metrics, respectively.CNN is a multilayer neural network that recognizes complex visual patterns which are extracted through a simple mechanism that is preprocessing the pixel images . As soonIn the proposed model, CNN is used for the patch-level inference whereas, at the WSI level, statistical methods are used to generate the final WSI prediction model for the proposed prediction system. The counting method was used in the pipeline approach as described above in detail. After all patches of one WSI are predicted by the CNN, we have collected and counted the prediction results of all patches obtained so far. Furthermore, the final WSI classification is the class with the most significant value in counting results.Displaying the significant feature regions of pathological images which are predicted by the proposed model can reveal the internal logic of CNNs and provide a further clinical reference about patient's data and health status. Therefore, our goal is to explore CNN's decision logic from the patch-level perspective and its accuracy in terms of predictions in the diagnosis process. Furthermore, it is highly likely that the proposed model's predictions are accurate and precise up to the acceptable level of doctors and patients. In the patch-level phase of the proposed prediction model, as shown in In this section is a comprehensive description of the various results obtained by applying the proposed system to various medical images (preferably benchmark in this case) and its effects on improving accuracy and precision of these systems. For this purpose, the proposed approach is thoroughly investigated using various images data collected by the Central South University Xiangya Hospital (CSUXH) during the aforementioned period of time. Likewise, a comparative analysis of the proposed scheme in terms of building trust of the concerned doctors and paramedical staff in the technologically generated diagnosis process is presented. These diagnoses are helpful to the practitioners and doctors in evaluation or examination of a particular patient in the healthcare systems.In the WSI-level melanoma and mole classification task, we have compared the performance of the proposed model with the results of at least 20 pathologists, i.e., manual examination and results. These experiments are carried out on the test dataset which is collected from the generated dataset of the Central South University Xiangya Hospital (CSUXH). Pathologists are able to freely view and understand all WSIs in the provided test dataset to verify its feasibility in the healthcare sector.In this figure, In order to explore the inherent logic of CNN diagnosis in the proposed model, we have used Grad-CAM to locate the significant feature areas of pathological images which is predicted by the proposed model. As shown in Figures In summary, the network has accurately locate lesion areas in a variety of complex situations. The activation map of patches indicated that the model could precisely detect lesion areas of melanoma or nevus. Furthermore, the ROI of the model agrees with that of pathologists.In the first row, the original melanoma patch with the lesion area marker (as red lines) is displayed. In the second row, the image is the activation map corresponding to the patch in the first row, and the red area represents the ROI of the model.In the first row, the original nevus patch with the lesion area marker (red line) is displayed. In the second row, the image is the activation map corresponding to the patch in the first row, and the red area represents the ROI of the model.We have reported a quantitative and scalable deep learning-enabled pipeline approach to identify melanoma and nevus using histopathology images in the smart healthcare. For diagnosis purposes, the proposed model has performed smartly and intelligently by providing the expected accuracy and precision in various decisions. The proposed model has outperformed the average pathologists on the melanoma classification tasks; that is, the accuracy of the proposed model is 93.3%, specificity 92.5%, and sensitivity 88.7%. Apart from this, the manual pathologist procedures and diagnosis are time consuming and costly  whereas the proposed model provided those judgments in comparatively minimum possible time intervals, that is, in seconds. Moreover, the result of the Grad-CAM method shows that the ROI of the proposed model overlaps with the lesion area.In the WSI classification task, the proposed pipeline-enabled diagnosis mechanism has achieved high accuracy and precision in terms of various decisions and prediction about the concerned disease and its classification in the real environment of healthcare application. The experimental results have verified the effectiveness of the proposed WSI diagnosis pipeline approach for the classification of melanoma and nevus. The proposed pipeline approach has mainly benefited from the powerful feature extraction capabilities of the deep learning method to guarantee classification of the pathology image data. We have observed water stains and staining differences in several WSIs of the proposed model. However, excitingly, it has not affected the proposed model outstanding performance in terms of various performance metrics such as accuracy, specificity, precision, and sensitivity on the available benchmark dataset which are available online.Apart from this, we have concluded that the Grad-CAM experiments are quite useful to precisely and accurately locate melanoma cells or nevus cells in the provided images data. The experimental results show that the diagnosis of the proposed model is not incomprehensible and is trustworthy. The model's focus on the lesion cell nest is greater than the collagen area, which shows that the model can effectively distinguish the lesion area from the nonlesion area. Similarly, the ROI of the model indicate that the diagnosis of CNN is also based on the lesion area.Furthermore, we have extended the classification mechanism of the proposed model to other common skin cancers and diseases with prognostic factors. We concluded that, by extending the visualization algorithm, the histological features learned by the proposed model have been fully displayed and help doctors further extract the potential histological features of melanoma. Moreover, studies have shown that additional clinical data can slightly increase the specificity and sensitivity of physician diagnosis. If other clinical data outside of pathological WSIs can be obtained during the clinical diagnostic process, those additional clinical data may also be helpful for model prediction in the deep learning approach.In this paper, we have developed a deep leaning and pipelining-enabled classification technique to assist pathologists and doctors in WSI diagnosis. Furthermore, the proposed model provides the diagnosis basis for a technological assisted mechanism with maximum possible accuracy and precision in terms of various decisions and predictions. Initially, a WSI diagnosis pipeline using a deep learning model and Grad-CAM is proposed to ensure feature extraction and classification of data. Secondly, we have collected 841 WSIs from Xiangya Hospital and built a large melanoma WSI dataset for model training and testing purposes. The proposed pipeline approach has the capacity to diagnose melanoma and provides visual evidence particularly in minimum possible time interval. Experimental results have verified that the proposed pipeline approach has outperformed manual pathologists diagnosis process particularly in terms of accuracy and precision. Furthermore, heat map has indicated that the proposed model accurately locates the lesion and histology features in WSIs and every evidence provided by the proposed pipeline is consistent with that of pathologists. In conclusion, the proposed pipeline approach helps the pathologists in diagnosis of the melanoma WSI and builds the trust in computer-assisted systems.In future, we are eager to extend the classification mechanism of the proposed model to other common skin cancers and diseases with prognostic factors. We believe that, by extending the visualization algorithm, the histological features learned by the proposed model will be fully displayed and help doctors further extract the potential histological features of melanoma.


Adapting to dynamic environments is essential for artificial agents, especially those aiming to communicate with people interactively. In this context, a social robot that adapts its behaviour to different users and proactively suggests their favourite activities may produce a more successful interaction. In this work, we describe how the autonomous decision-making system embedded in our social robot Mini can produce a personalised interactive communication experience by considering the preferences of the user the robot interacts with. We compared the performance of Top Label as Class and Ranking by Pairwise Comparison, two promising algorithms in the area, to find the one that best predicts the user preferences. Although both algorithms provide robust results in preference prediction, we decided to integrate Ranking by Pairwise Comparison since it provides better estimations. The method proposed in this contribution allows the autonomous decision-making system of the robot to work on different modes, balancing activity exploration with the selection of the favourite entertaining activities. The operation of the preference learning system is shown in three real case studies where the decision-making system works differently depending on the user the robot is facing. Then, we conducted a human–robot interaction experiment to investigate whether the robot users perceive the personalised selection of activities more appropriate than selecting the activities at random. The results show how the study participants found the personalised activity selection more appropriate, improving their likeability towards the robot and how intelligent they perceive the system. query Please check the edit made in the article title. Social robots with autonomous decision-making capabilities are becoming real in many applications. They are used in tasks related to healthcare . In this work, the input vector is the features of the user. The output is the ranking of activities according to the preferences of the user. The goal of LR is to learn a mapping from mentclass2pt{minimThe integration of random forest classifiers Breiman  in LR alRanking by Pairwise Comparison  and four RGB LEDs . It also has a speaker to communicate and play sounds verbally. Mini communicates with an external touch screen for displaying different games or multimedia content, among other visual and auditory information.The sensorimotor system of Mini consists of a broad range of sensors and actuators to interact with the environment. Mini contains a 3Figure The human–robot interaction system  to control the robot’s actuation. It checks the state of each actuator and controls whether it is possible to execute specific expressions using a priority system. On the other hand, this system receives high-level expressions from the expression scheduler, decomposing them into individual commands to each robot actuator. A particular player dedicates to each actuation unit managing the control of the physical actuator. Finally, the expression system returns essential information to the human–robot interaction system about the execution of each expression.Finally, the decision-making system is the most critical part of our architecture. It appears highlighted in Fig. Play Bingo! game: This activity allows the robot to play the well-known Bingo! game with the user.Play a calculus game: This game consists of the robot asking the user to solve some mathematical operations with different difficulty levels.Play a quiz game: This game presents the user a set of questions about different topics , and the user has to guess the correct answer to win.Weather forecast: This activity forecasts today’s weather prediction.Informing news: The robot informs the user about recent news related to some fields as sports, national events, international events, opinion articles, and last-minute information.Play music: The robot plays songs about a predefined music style (which can be decided by the user or the robot). The available music styles are English pop, English rock, Spanish pop, Spanish rock, Latin music, Classical music, and Flamenco.Display videos: The robot displays a video using its tablet. The available categories of the video are Sports, Cooking, Famous film trailers, and Comedy.Play audiobooks: The robot plays an audiobook of the following categories: Famous historical moments, classical books or children’s tales.Play sayings: The robot tells the user a well-known Spanish saying.Play jokes: The robot tells the user a funny joke about different topics.Display photos: The robot displays a group of different photos, which can be about Historical monuments, Incredible landscapes, Animals, or Funny moments.The following list introduces and describes the entertaining activities of the social robot Mini. The preference learning system uses these activities to personalise the interaction suggesting the users their preferred ones.The adaptive human–robot interaction carried out by the preference learning system is summarised in three steps. First, the entertaining activities of the robot are classified into categories and subcategories. Second, LRF techniques rank the items in each category and subcategory to generate the user preferences. The decision-making system uses these rankings (and associated scores) to carry out a three-step hierarchical decision. The third step embraces the decision itself, leading to the execution of a particular entertaining activity that will usually be among the user’s preferred ones. The decision consists of selecting a category, then a subcategory, and finally an entertaining activity. The entertaining activity must be contained in the subcategory and category selected previously. Instead of directly selecting an activity from the available list, we propose a hierarchical selection that allows the decision-making system to work in different modes.Figure Another example is the Videos dataset, which sorts the entertaining activities related to videos displaying: comedy, sports, film trailers, and cooking. Note that like Fig. Using the ranking defined by each dataset, the decision-making system decides which activity to execute deciding level. The decision-making system works in three modes. These modes combine autonomous decisions made by the robot with decisions from the user. The operating mode depends on the user proactivity level, a parameter gathered by the list of features of the user. The proactivity level can have five numerical values, from 1 to 5. Proactivity values close to 5 represent a user that typically takes the initiative while interacting. Contrarily, proactivity values close to 1 unit indicate a user not taking the initiative while interacting with the robot. The decision-making system works autonomously with users with low proactivity, allows them to select the entertaining activity to execute when encountering very proactive users, and balances both modes for moderate proactive users. We believe that adapting to the proactive level of interacting improves the interaction since the robot aids them if they have difficulties using it.p(a) to each alternative. The selection probability depends on the score yielded by the preference learning system (R(a)), the scores of the other N activities, and a parameter in the Boltzmann equation called temperature , and finally, the entertaining activity executed. Therefore, options with similar scores will present similar selection probabilities. The options with higher scores are more likely to be selected. Section The second mode consists of the robot giving the initiative to the user so (s)he can decide which entertaining activity to execute. This mode activates when the robot encounters a very proactive user. The user selects the category, subcategory and entertaining activity using the tablet menus shown in Fig. The third mode combines modes one and two. This mode activates with users with moderate proactivity. In this mode, the robot combines autonomous decisions with giving the selection to the user. Thus, we allow the users to explore different alternatives from the repertoire of activities rather than leaving the entire decision in the robot’s hands. Using this method, we avoid the robot making similar decisions. Section The Multimedia category divides into Videos, Photos, Audiobooks, Sayings, Jokes, and Music subcategories. Each subcategory contains a group of activities at the lowest level of Fig. This section describes how to endow the social robot Mini with preference prediction for personalising its entertainment activity selection. In the first place, we present the process of building the datasets used for estimating the preferences from data retrieved using an online survey. Then, we finally explain how both LR algorithms have been compared to select the ones with the best performance. Finally, we describe the human–robot interaction experiment to validate whether the users prefer a personalised activity selection instead of a random one.The purpose of this work aims at predicting user preferences and use them to achieve a personalise activity selection. Like similar studies , Audiobooks , Photos  and Videos . Each category and subcategory defines a dataset that contains a particular ranking for each user. Using the ratings provided by each participant, activities with higher ratings are ranked on top. Rating ties were randomly broken. Using the activities classification in categories and subcategories and the rankings, we built 8 datasets. The first dataset ranked the categories . The other datasets ranked the activities in the categories Games, Information, and Multimedia and the subcategories Photos, Videos, Audiobooks, and Music. Moreover, we built 2 additional datasets, Entertainment and Multimedia. This dataset organisation allows the decision-making system to select a category hierarchically, then a subcategory, and finally an entertainment activity that will be executed. Thus, all datasets are necessary to personalise the activity selection since the ranking they contain indicates which categories, subcategories, and entertainment activities each user likes the most and will be selected more often.To define the labelled output of the dataset, the survey requested users to rate how much they like the entertainment activities of the robot. Ratings ranged from We received 473 replies to the online survey, which led to 471 input instances since three replies were not valid due to inconsistencies in the features . The participants were 217 men, 254 women aged Preference learning predictions are aggregated rankings that represent the preferences of an instance  Kendall  and SpeaSpearman  nonparamKendall’sb  have multiple parameters. We opted for using the node splitting criteria . We decided to include customised questions considering previous similar works in the evaluation of social robots , Intelligence perceived (IP), and to a lesser extent Animacy (AN) are those more related to adaptive and personalised activity selection, so we believed that participants of Condition 1 would perceive the robot exhibiting a personalised activity selection prominent in these attributes. However, we did not expect to obtain differences in the Anthropomorphism (A) and the Security Perceived (SP) attributes as both are related to the robot’s physical features without any influence on personalised activity selection.The following section presents the results obtained in selecting the best preference learning algorithm regarding the hyperparameters optimisation, the prediction accuracy of random forest and ranking metrics. Once we selected the best algorithm, we describe three case studies showing the operation modes of the decision-making system during the recommending the preferred entertainment activities of the robot. Finally, we conclude by presenting the results of the preference learning system performance in real human–robot interactions. Figure The selection of the LRF algorithm that produces the best outcomes in our multi-dataset hierarchical learning system led to comparing Ranking by Pairwise Comparison and Top Label as Class, two outstanding methods with different features. Next, we show the comparison results and select the LRF method more appropriate for our model.Figure Using the results provided by Fig. Table Results show how Ranking by Pairwise Comparison produces better classifications than Top Label as Class for all cases. Moreover, Gini impurity usually presents better results than Information gain. Focusing on the numerical results of the classification, the high accuracies in the Games, Multimedia and Information datasets are remarkable, all above Table m is the number of items to rank. Although Top Label as Class trains one model per dataset, the number of items to rank also affects the training time, but to a lesser extent. Finally, the comparison of Gini impurity with Information gain presents very similar values, not showing any significant difference.Ranking by Pairwise Comparison requires additional computational time as it trains Table Concentrating on Kendall’s Table entclass1pt{minimaentclass1pt{minimaConsidering the results obtained from the comparison of Ranking by Pairwise Comparison and Top Label as Class, we opted for selecting Ranking by Pairwise Comparison since it generally provided better estimations. As results in Table As we expected, Ranking by Pairwise Comparison provides substantially better results for the Games and Multimedia datasets than Top Label as Class, probably because their dataset definition ballast the replacement of the top-label carried out by Top Label as Class. Although in the other datasets Ranking by Pairwise Comparison still outperforms Top Label as Class, in these cases the differences are minimal. Remember that these results arise from the hyperparameter optimisation method described in Fig. The following section presents three case studies describing Mini’s autonomous entertainment activity selection using the user preferences and the level of proactivity. These cases correspond to the three operation modes of the decision-making system explained in Sect. This case of study shows how the preference learning system presented in this work produces different preference estimations for two different users. Thus, the decision made by the decision-making system of the robot adapts to the estimated preferences of each user, producing a personalised interaction. Table The predictions for User 1 lead the decision-making to select more often specific entertainment activities. These selections depend on the temperature values set in the Boltzmann Eq. Considering the features of User 2 in Table This autonomous operation mode presents the advantage of proactively start the interaction with inexperienced users or those with interaction limitations. Thus, the robot fosters interacting with the user engaging him/her with entertainment activities. On the other hand, always selecting the activities without the intervention of the user may lead to repeat the preferred activities provoking the user’s fatigue. Besides, the estimations produced by the preference learning system may not be the real preferences of the user, so this method would require a complementary adaptive refinement of the initial predictions with the interaction.This case of study shows the second operation mode of the decision-making system. This mode activates when the robot interacts with users with a high level of proactivity. In this mode, the robot guides the user during the entertainment activity selection process, but the user decides which activity the robot will finally execute. Like in the previous mode, the activity selection follows a three-level hierarchical path. First, the user has to select the category, then the category, and finally the entertainment activity that (s)he prefers. A sequence of tablet menus is displayed on the touch screen to assist the user in selecting and letting him/her know about what activities it has. Figure An example of this operation mode could be the following. Initially, the Entertainment menu will be permanently displayed, ranking the categories Games, Multimedia, and Information Fig. a accordiContrary to the first operation mode, this method has the advantage of allowing users to select their favourite entertainment activity, so the probabilities of making an incorrect selection are substantially reduced. This method is useful for adapting the initial predictions with the interaction as users are providing positive feedback by selecting their preferred activities. However, with users that do not like the robot and refuse to interact with it, this mode can lead the user to stop the interaction since (s)he can feel that the robot behaves like a simple automaton.The third case of study presents how the decision-making system of the robot balances autonomous decisions with giving the initiative to the user. This operation mode activates for users with a moderate proactivity level. Considering the feature of User 3 in Table An example of operation mode that considers the rankings and scores predicted for User 3 shown in Fig. Considering the case studies presented in this section about how the decision-making system of Mini works, the level of exploration (in terms of executing a broader range of different activities) depends on two factors. First, it depends on the temperature of the Boltzmann equation when the robot autonomously decides what item to select at each decision stage using estimated preferences for a particular user. Additionally, the exploration is also higher when more decisions depend on the user, as (s)he can decide the real preferences (which can be different from the estimated ones). Although allowing the user to select by (him)herself what activity the robot will execute may promote exploration, autonomous decision-making becomes especially necessary for users with interaction limitations and low proactivity.From our point of view, this operation mode will provide the best results in the long run. We hypothesise that the combination of the robot’s autonomous decisions with user selection could reduce the fatigue of the user and increase the engagement. Besides, balancing the activity selection may drive the user to test a broader repertoire of entertainment activities, discovering their real favourite ones. A possible drawback of this method can be related to the expectations of the user. If (s)he is waiting for a menu to appear and the robot decides a different activity, the user may feel that the robot is not really adapting to him/her. For this reason, it is important that autonomous and user selection are correctly balanced to avoid these situations.Personalisation perceived reported by the participants of Condition 1  and Condition 2 .The evaluation of the system during human–robot interaction provided valuable results regarding both conditions tested. Figure The statistical analysis of the Personalisation perceived, presented in Table The analysis of the results related to the Godspeed questionnaire lead to the following conclusions. As Fig. p-valuep-value=0.01) and no differences in the categories Anthropomorphism (A)  and Security Perceived (SP) . These results align with our initial hypothesis, indicating that the robot with personalised activity selection (Condition 1) is rated more positively than a robot with random activity selection (Condition 2). Contrary to what we expected, we did not find any significant statistical differences in Animacy (AN) .The statistical analysis of the data also provided valuable results in the comparison of both conditions. Since the data retrieved in the evaluation followed a normal distribution, we could apply parametric statistics. In our experiment, we have two independent samples (Condition 1 and Condition 2). For this reason, the most appropriate statistical analysis to investigate whether there exists a significant difference between them is the Student’s T-test Efron . Table 5One of the goals of social robots is to communicate with their users appropriately. These users may present very different features, including a diverse range of preferences. For this reason, if a robot aims at entertaining users during long-term interactions, it has to exhibit adaptive capabilities to suggest each user with their favourite activities. Currently, many machine learning algorithms allow fulfilling this goal. However, like presented in this manuscript, preference learning and, more precisely, LRF is nowadays a competitive alternative that allows artificial systems to predict, using rankings, the favourite activities of each user from a predefined set of alternatives. The review of the existing literature lacks research about this area, as most current models focus on learning while interacting instead of anticipating the potential preferences of users before the interaction occurs using their defining features. However, we took strong inspiration from the literature presented in Sect. Due to the broad range of algorithms that provide meaningful results in LRF problems, we opted for comparing two promising but different alternatives: Top Label as Class and Ranking by Pairwise Comparison. We demonstrated how Ranking by Pairwise Comparison is the best alternative since it produces promising outcomes in our learning scenario, especially for datasets with a large number of items to rank. To prove the improvements of personalising the activity selection according to the user’s preferences, we present different case studies about the operation of the learning system in the long run. Besides, we conducted a short-term human–robot interaction experiment for assessing whether people prefer interacting with a robot presenting personalised activity selection (Condition 1) than with a robot with random activity selection (Condition 2). The statistical analysis carried out on the data retrieved from the study shows valuable outcomes in comparing both conditions. The participants believe that a robot with personalised activity selection is more appropriate to their preferences, as well as they perceive this robot as more intelligent, and they like it above the robot without personalised activity selection.Although the previous results show optimistic and promising outcomes in the adaptive selection of activities by the robot and the correct operation of the system, the learning process of the robot has some limitations that are worthy of mentioning. In the first place, we are aware that retrieving the data using an online survey for building our datasets and training the model is not the best alternative. Nonetheless, due to the COVID-19 pandemic, we believed it was the safest way to obtain the necessary data. Besides, the collection of the input features for test users was carried out using the questionnaire presented in Appendix Another limitation of the system resides in comparing the two LRF algorithms used in this work. We selected them because previous preference learning scenarios suggested promising results, but many other alternatives could be selected. Besides, as Sect. The human–robot interaction experiment yielded encouraging results in evaluating the robot exhibiting a personalised activity selection. However, we are conscious that further experiments with more participants and during long-term interactions are required to see if the learning model works in the long run. The system presented in this work is just the first step in endowing social robots with personalised activity selection. In the future, we believe that the preference learning estimations should be combined with reinforcement learning to adapt the initial predictions to their real values using the feedback provided by the users. Finally, we pretend to enlarge the number of instances of the datasets by including the final ratings learnt by the reinforcement learning model to enhance the prediction accuracy and the estimated rankings of the original preference learning model presented in this dissertation.This contribution presents how the social robot Mini can predict the favourite activities of the potential robot users. Specifically, the adaptive decision-making system extends our previous research focused on using the robot during personalised interactions. First, we compared the performance of Ranking by Pairwise Comparison and Top Label as Class, two encouraging LR algorithms. The results demonstrate how Ranking by Pairwise Comparison outperforms Top Label as Class in our learning system, producing better overall estimations. Then, we present different case studies showing the operation of the architecture. Finally, a human–robot interaction study validates whether people prefer a personalised activity selection over a random one.The preference learning system embedded in the decision-making of Mini includes different mechanisms of interaction depending on the user with whom it is interacting. The three case studies describe how the robot interacts with users with different proactivity levels, leading to broad and diverse communication between agents. Moreover, the decision-making system promotes activity exploration and user proactivity if necessary. In this context, we are aware that predictions, in certain situations, do not assure a perfect accuracy in preferences representation, especially whether users have not interacted previously with the robot or do not figure out how some of the activities work. Besides, sometimes users can change their interests in real time while interacting with the robot. To overcome these situations, this work embraces the first step towards preference learning modelling in social robots.


Multimode fibers are attractive for imaging, communication, computation, and energy delivery. Unfortunately, intermodal and polarization coupling precludes direct control of the delivered mode composition. We present a technique to tailor the mode composition at the output of a multimode fiber with thousands of modes, which we refer to as myriad-mode fiber, using its experimentally measured transmission matrix. While precise mode control has been demonstrated in typical multimode fibers with up to 210 modes, the method proposed here is particularly useful for high mode number fibers, such as when the number of modes is comparable to the number of modes of the wavefront shaping spatial light modulator. To illustrate the technique, we select different subsets of modes to create focal spots at the output of a fiber with 7140 modes. Importantly, we define efficiency and fidelity metrics to evaluate the mode control and demonstrate the relationship between efficiency, fidelity, and the spatial location of the spots across the distal fiber cross-section. I.MULTIMODE fibers (MMFs) have found application in classical , 2], ,  and quanAll these applications are enabled by some form of control over the modes of the fiber. Recently, spatial light modulators (SLMs) have been used for controlling all the modes in the fiber mode basis of typical multimode fibers with up to 210 modes , 30], , . HoweverSpecifically, in the field of fiber-optic communication, MMFs hold significant interest due to their large information carrying capacity. The achievable data rates through MMF transmission systems however are still severely limited by modal dispersion, coupling and nonlinearities. Mode division multiplexing is a promising technique which allows using different modes of the MMF as separate information channels to enhance the fiber capacity. It has been demonstrated by offset launching , using pIn this report, we present a method to select specific groups of fiber modes at the output of an MyF using a phase-only spatial light modulator at its input. As an application example, the selected modes are constructively interfered at a predefined location in the fiber output, hence forming a focal spot. While we use the Hadamard basis at the input and generate focal spots at the output, we achieve mode control in the fiber mode basis via a digitally implemented basis transformation. As a result, one can take advantage of the different properties of mode groups and their interactions. The fiber mode composition of the focal spot can be tuned by modifying the input pattern. Furthermore, we perform experiments to compare the sensitivity of focal spots to fiber bending when created using two different complementary mode sets. Towards a quantitative evaluation of the quality of mode control, we define specific efficiency and fidelity metrics that help understand the possibilities and limitations of mode control.II.Wavefront shaping is becoming a key technique in imaging and energy delivery through scattering media and MMFs. One of the preferred approaches involves characterization of the medium, in our case the fiber, through the transmission matrix (TM) , 29]. T. T29]. TEtarget, at the distal end of the fiber. Etarget can be written as a superposition of all the fiber modes. Assuming a radially symmetric parabolic refractive index profile, the linearly polarized (LP) modes of a GRIN (graded-index) MyF can be represented using the Laguerre-Gauss (LG) field solutions . W. WEtargeNm number of modes per polarization, stored in the columns of a matrix, F, which we call the mode matrix, we can represent an Nout- pixel Etarget field in the modal basis by taking its product with the inverse of the F matrix. However, F can only be invertible when it is a square matrix, which is true only when the number of samples in Etarget is equal to the number of fiber modes. In the experiments described below, we over-sample each speckle grain appearing at the output to maintain a good SNR, which leads to a highly rectangular and non-invertible F matrix. Hence, for the general case, we find the modal representation of Etarget using the Moore-Penrose pseudo-inverse of F, F+ as described in For an MyF with Mtarget is Nm × 1 vector of mode coefficients corresponding to the vectorized 2D field Etarget , F is the Nout× Nm mode matrix and F+ denotes its pseudo-inverse. In order to tune the mode composition of the target field, we find the orthogonal projection of Etarget, F′ as shown in F′+ denotes the pseudoinverse of F′ defined as F′+ = (F′†F′)−1F′† and † denotes the conjugate transpose. In physical terms, Etarget that can be generated with the selected Nm′ modes.Here, Kobs is then used to generate the mode tailored field Ein calculated using The measured complex-valued TM of the fiber, η, is defined as the ratio of total energy in the selected fiber modes and the sum total energy in all the modes as shown in We define two metrics to evaluate the mode control performance at the output fields, efficiency and fidelity. The efficiency, denoted by Eout, which were calculated by back-projecting the output field, Eout on the fiber modes basis by multiplying it with F′F′+ and F″F″+ respectively. The fidelity on the other hand, denoted by C, is defined as the Pearson correlation coefficient between the target and experimentally obtained intensities as defined in In the equation above, the single and double primes denote the selected and non-selected mode components of the experimental output field σ denoted the standard deviation of the variable in the subscript. While the efficiency characterizes the confinement of energy in the selected modes, the fidelity characterizes the spatial control ability.Here, COV denotes the covariance function and III.The experimental setup used in our experiments is illustrated in μm (Newport F-MLD) for all experiments. We place a linear polarizer at the distal tip to limit the TM measurement to a single polarization. In the absence of polarization coupling, this would mean that the 4096 independent phase pixels of the DMD control 3570 single polarization modes of the MyF. However graded index fibers show significant polarization coupling ..μm (NewpO(NinNout). Each generated focal spot fields, Eout was measured using three phase measurements, just as done during calibration and their corresponding mode coefficient vectors, Mout are determined using Eout instead of Etarget.Different sets of basis functions can be chosen to measure the TM, including canonical plane waves or focal spots at the input facet of the fiber. For this study, we chose the Hadamard phase basis because of its ease of implementation with the DMD. The calibration required 12288 measurements which were made in about three minutes. The calculation of the pseudoinverse of the mode matrix has a complexity of emanding . The optIV.F. A first set of scanning spots was created with the half lowest order modes (LOMs) in F and the second set of spots was created using the half highest order modes (HOMs). Eout for the two focal spots and Mout. We can observe that while the mode coefficients of the non-selected modes cannot be completely suppressed in the experiment, the mode coefficients of the selected modes are in good agreement with their targeted values.To demonstrate mode tunability, we created focal spots using two subsets of modes in the mode group-ordered mode matrix The high efficiency and fidelity values obtained for the focal spots generated with selected modes are indicated in their respective sub-figures in Interestingly, when focusing in the near field of the distal end of the fiber, the mode efficiency varies with the radial location of the focal spot when a given HOM or LOM set is selected. This is because different modes have different spatial support and are more or less suitable for the target output. To demonstrate this, we show plots of efficiency of mode-controlled focal spots created at increasing distance from the center of the fiber using LOMs and HOMs respectively  were computed as the product of the input fields Ein found in Kobs. The F′ containing only LOMs for 5 (i) (with mode control). The LOMs and HOMs mode components of each of the above output fields [5 and 5] were found by back-projecting the output fields, Eout on the fiber modes basis by multiplying them with F′F′+, where F is the fiber mode matrix with the LOMs or HOMs in its columns. The HOMs component is non-zero even when LOMs are chosen to create the spot field because the mode control efficiency is not 100 %.In general, although the efficiency and fidelity values can be lower outside the optimal region of mode sets, the proportion of energy in the selected modes improves significantly due to mode control. To visualize this improvement, we show the digitally computed output fields and their LOM and HOM components for a focal spot created without and with mode control i.e., using all the fiber modes and using LOMs . F. F52]. FAs opposed to prior work, efficient mode control in MyF is much more challenging due to the limited number of degrees of freedom provided by the DMD and the inherent complexity of the system. However, the approach ensures that maximum energy is confined to the selected modes at the output and although all the fiber modes still propagate to the distal end of the fiber, the focal spot itself, which is many times brighter than the background, is primarily a result of the interaction of the selected modes. It should also be emphasized, that the technique controls the mode composition at the output of the fiber, which is not necessarily the same as the mode composition throughout propagation due to mode coupling resulting from perturbations of the fiber . Because the mode composition throughout the fiber is complex, our technique is more suitable for generating complex mode combinations rather than a combination of fewer modes or a pure mode, unlike other traditional mode control techniques.The proposed method also provides an avenue for combining the advantage of large core MyFs for a larger bandwidth, higher NA and bigger field of view, with the bend resilience of an MMF with fewer modes. The extent of intermodal coupling or bend sensitivity of an MMF is inversely proportional to the difference between the propagation constants of the modes . Hence, The idea of mode selection through few-mode MMFs is already a topic of wide interest in the field of fiber-optic communication. Mode selection in MyF, could allow the use of groups of modes with similar dispersion and delay profiles as different channels, where the number of channels can be smaller than the total number of modes in the fiber. Other possible applications of mode selection in MyF include control of individual mode groups for spatio-temporal focusing, quantum communication, and energy delivery.supp1-3057615


Two important tasks in many e-commerce applications are identity verification of the user accessing the system and determining the level of rights that the user has for accessing and manipulating system’s resources. The performance of these tasks is directly dependent on the certainty of establishing the identity of the user. The main research focus of this paper is user identity verification approach based on voice recognition techniques. The paper presents research results connected to the usage of open-source speaker recognition technologies in e-commerce applications with an emphasis on evaluating the performance of the algorithms they use. Four open-source speaker recognition solutions  have been evaluated in cases of mismatched conditions during training and recognition phases. In practice, mismatched conditions are influenced by various lengths of spoken sentences, different types of recording devices, and the usage of different languages in training and recognition phases. All tests conducted in this research were performed in laboratory conditions using the specially designed framework for multimodal biometrics. The obtained results show consistency with the findings of recent research which proves that i-vectors and solutions based on probabilistic linear discriminant analysis (PLDA) continue to be the dominant speaker recognition approaches for text-independent tasks. In a world where Internet services have become the backbone of Internet usage, identity management systems are an important part of everyday life. Automatic speaker-recognition systems, as part of identity management in many e-commerce applications, in addition to general business interactions, have emerged as a pertinent means of identity verification . Two impThe choice of biometric modality depends primarily on the nature of the application, the access device, the conditions in which the application is used, the required decision reliability, and the modality characteristics. For example, in mobile and home banking, transactions should be carried out using a mobile network, or the Internet, so it is convenient to use the human voice as one of the factors for user verification . The neeWith the increased interest in the use of voice as a method for user identification, the aim of this paper was to research the impact that a change in working conditions in the process of voice recognition has on the performance of identity verification and access control systems.The human voice contains both physical and behavioral biometric characteristics . As withEven though voice recognition is a practical method for identity verification, it faces various difficulties, such as a variation of speakers’ vocal tract characteristics, emotional states, traits of the microphones used, a noisy environment, and the quality of the communication channel between the time of enrollment and future recognition.More robust models for speaker identification, non-matching enrollment, and recognition conditions have been published ,8. BasedHowever, localization of an e-commerce application with voice-based user identification may further downgrade the performance of the system. In a case where the user communicates with the application in a foreign language, it is necessary to examine how this affects the performance of the speaker recognition system because localization of the application has not been carried out ,10.This paper presents an evaluation of the performance of four popular open-source speaker recognition solutions  of the various enrollment and recognition conditions. Mismatched conditions are influenced by the varying lengths of spoken sentences, types of recording devices, and the differences between training and operating languages. All the tests were performed under laboratory conditions using the specially developed framework for multimodal biometrics . The maiThe next chapter gives an overview of the state of the art in the area of speaker recognition. In chapter three, the most popular open-source solutions for speaker recognition are presented. The following chapter describes the setup of the experiment performed as part of this research, with the results presented and discussed in the next section. Finally, conclusions are given based on the discussion and results.The speaker recognition process uses essential speech information to claim the speaker’s identity. The fundamental components of any speaker recognition system are the speaker models and speaker feature extraction algorithms. The speaker model faithfully represents a speaker and is created in the system based on the speech feature vectors. Feature vectors are derived from the input speech by applying one of the extraction algorithms.Text-dependent speaker recognition;Text-independent speaker recognition.In the enrollment mode, a speaker model is trained using the feature set derived from the utterance of the target speaker . DiffereGaussian mixture models (GMMs);Neural networks.For each of the speaker modelling approaches, the relevant literature suggests applying different modelling techniques. In text-independent speaker recognition, popular modelling techniques include:Using Gaussian mixture models is not a new technique, but it is very efficient when it comes to speaker recognition. GMMs are used in many applications that are involved in statistical data analysis, pattern recognition, computer vision, voice and image processing, and machine learning .Neural networks are a new approach in speaker recognition, and they are starting to take precedence in the field. Several types of neural networks are used in the speaker recognition process. Richardson et al. concluded that deep neural networks (DNNs) are very effective in speaker recognition . VillalbNeural networks speaker modelling approaches are widely implemented and used in commercial applications. IDVoice, a product of the ID R&D company, with headquarters in the United States, combines proprietary x-vector technology, deep neural networks (DNN), convolutional neural networks (CNN), and their own patented p-vector technology to achieve exceptional performance . VoiSentWhen it comes to text-dependent speaker recognition, the best practices are systems based on hidden Markov models. The traditional hidden Markov model (HMM)-based systems give DNNs comparable results in terms of speaker recognition accuracy. Unfortunately, DNNs require a large training sample .de facto standards for evaluating the performance of a speaker recognition system. Two parameters widely used for the evaluation of biometric recognition systems are recognition accuracy and recognition speed. Accuracy can be quantitatively described by the following measures:False acceptance rate—FAR;False rejection rate—FRR;Equal error rate—EER.It is important to describe the parameters which are known as The false acceptance rate (FAR) measures the likelihood that the biometric system, based on the calculated score and chosen threshold, inaccurately decides that the input feature set and the feature set selected from the database belong to the same person. The false rejection rate (FRR) measures the likelihood that the biometric system, based on the calculated score and chosen threshold, inaccurately decides that the input feature set and the feature set selected from the database do not belong to the same person. The equal error rate (EER) refers to a specific operating point of the biometric system where the FAR equals the FRR. It should be noted that the biometric system with the lowest EER value has the best decision-making performance .In the reference literature, there are various open-source speaker recognition solutions. Each of them follows at least one speaker modelling approach and implements at least one speaker feature extraction algorithm. Some of the most popular open-source speaker recognition software packages and toolkits include SPEAR, MARF, ALIZE and HTK. Different approaches to speaker recognition problem-solving in those solutions allows evaluation of, not only the performance under the same operating conditions, but also the evaluation of algorithm sensitivity to the variations in parameter values that define those operating conditions.SPEAR is an open-source solution and contains an extended set of speaker recognition tools. It is based on Bob Toolkit, which contains free signal processing and machine learning components. The toolkit supports several modelling techniques: Gaussian mixture models (GMMs), intra-session variability (ISV), common factor analysis, and total variability (i-vectors). It is developed in C++, but is also provided in the Python environment . SPEAR aMARF (Modular Audio Recognition Framework) is an open-source platform that contains a collection of voice, sound, speech, text, and natural language processing algorithms written in Java. MARF includes several feature extraction algorithms such as fast Fourier transform (FFT), linear predictive coding (LPC), artificial neural network (ANN), as well as various distance classifiers . In termThe ALIZE open-source toolkit for speech recognition is based on Gaussian mixture modeling with a universal background model (GMM-UBM). It supports state-of-the-art techniques, such as: joint factor analysis (JFA), support vector machine (SVM), i-vector modeling, and probabilistic linear discriminant analysis (PLDA) . This toHTK (HMM Toolkit) is a toolkit generally used to build and model hidden Markov models. In the HMM-based speech recognition system, each word is represented by speech vectors. It is assumed that a Markov model generates the sequence of observed speech vectors corresponding to each word. This toolkit consists of training and recognition processes. Specifically, it supports several sub-steps: data preparation, training, and recognition .Considering the popularity of open-source solutions, as well as the different algorithms they implement for extracting speaker features, the authors of this research have selected them as perfect candidates to compare and evaluate the performance of speaker recognition systems in different environments with varying parameters. The setup of this experiment and its specifics are described in detail in the next chapter.The enrollment phase;The recognition phase.The main goal of the experiment was to examine how the mismatch between enrollment and recognition conditions influences the performance of the speaker recognition system. Thus, the experiment was performed in two phases:During the enrollment phase, the valid user’s speech data were collected for setting up the selected speaker recognition models. Subjects spoke the sentences following the provided text instructions, thus representing valid users of the system. The collected speech data were recorded in Serbian and English. Serbian was chosen because existing works lack research which include this language, whereas English was chosen as the language most used in similar speech and speaker recognition experiments.In the recognition phase, training data were compared to new input voice data in order to evaluate the performance of the speaker recognition system.There was a total of 40 subjects, of which 30 represented valid users and enrolled the training data. Training data were gathered from the subjects in laboratory conditions. The other 10 subjects were representing imposters and provided data during the recognition phase. All of the subjects fell under the age group ranging between 18 and 35 years. Out of 30 valid users, 16 were female and 14 were male.An integrated microphone on a laptop;An external microphone on a desktop computer;A 3G mobile phone.During the data acquisition, both in the enrollment and recognition phases, the subjects were simultaneously recorded with three devices:Detailed specification of the used recording devices is presented in Subjects were fluent in Serbian and English, and were recorded in both languages. Textual instructions in both Serbian and English were created for recording purposes.Short (up to 1 s);Medium (1 to 2 s);Long (over 2 s).Text prompts were separated by length and by context. There were three lengths:The first set of statements contained the voices of characters, numbers, and basic commands that simulated interaction with the operating system and the menu;The second group of sentences utilized terms usually found in the banking vocabulary, for instance, “Do you offer cash loans indexed in Serbian Dinars?”;The third group of sentences consisted of terms used in everyday life situations, such as the names of beverages, food, the time of day, common actions, etc.Text statements were divided by context into three categories:All words and sentences were collected in both Serbian and English. By using the collected data of the valid users, each chosen speaker recognition model was trained separately in English and Serbian.Testing and evaluation of the system was performed in the recognition phase by using valid users’ utterance, as well as 10 additional new users,’ which imitated the role of imposters.Before presenting the results of our research, we will first give a brief overview of a possible way to evaluate the performance of a biometric system.The efficiency of the biometric system is reflected in the accuracy and speed of decision making. The calculation of the similarity score between the two specified feature sets is the basis for deciding whether or not both data sets belong to the same entity. If the calculated score is above the chosen value, named as the threshold, we believe that both feature sets belong to the same entity. We say that both feature sets are a case of intra-class variation. Of course, great attention should also be paid to the speed at which the scores are made. Due to the many comparisons performed, slower methods should not be used in the identification mode.English in both phases;English in enrollment and Serbian in the recognition phase;Serbian in enrollment and English in the recognition phase;Serbian in both phases.To test the sensitivity of the open-source speaker recognition solution to language mismatch, both English and Serbian were used as different languages in the enrollment and recognition phases. There were a total of four scenarios using the enrollment and recognition steps:For each scenario, measurements were made for nine different working conditions, determined by the length of the spoken text  and the used recording device .The performance of the 144 tested configurations with open-source solutions, measured by the equal error rate (EER), are shown in The best performances of the speaker verification system grouped by solution are shown in When using SPEAR, it is notable that using Serbian language in enrollment and English in the testing phase results in a slightly higher EER percentage. Similar conclusions can be drawn for the case where MARF is used.The ALIZE open-source toolkit for speaker recognition showed the best performance in all language combinations. Speaker recognition software tool, SPEAR, came in second place, and MARF’s open-source platform came in third. The weakest speaker recognition scores were obtained using the HMM Toolkit, HTK;All four tested open-source solutions gave the best results with a combination of English as both a training and test language. Somewhat weaker results were obtained in cases where Serbian was used as a training and test language. In these cases, the EER increased from 9% for MARF to 28% for ALIZE.By analyzing all of the obtained results, the following conclusions can be drawn regarding the selected open-source solutions:It is interesting to note that, in the case of a mismatch between a training and test language, better results were measured for the combination of English as a training language and Serbian as a test language. The EER values for the configuration of Serbian as a training language and English as a test language increased from 27% for SPEAR to 70% for ALIZE, compared to a combination of English as a training language and Serbian as a test language.In all cases, regardless of the open-source solution used, the language, and the voice recording device, the ERR values decrease monotonically with the length of the test sequence. Therefore, the best performance of the speaker recognition system is obtained for the longest test sequence. By extending the sentences from short records (up to 1 s) to long records (over 2 s), the EER decreased by an average of about three times.The EER values shown in This research aimed to determine how the performance of a system for identity verification and access control, based on speaker recognition techniques, is influenced by chosen open-source technologies, various lengths of spoken sentences, types of recording devices, and mismatch between a training and an operating/test language. This study examined the performances of four popular speaker recognition system solutions that implement different approaches and techniques to the problem under consideration. After analyzing the results of the speaker recognition system, obtained in 144 different tested configurations, the conclusion was that the most successful open-source speaker recognition solution is ALIZE. The ALIZE open-source toolkit for speaker recognition is based on Gaussian mixture modeling with a universal background model (GMM-UBM). It supports techniques with i-vector modeling and probabilistic linear discriminant analysis (PLDA). The results obtained with the ALIZE and SPEAR packages are consistent with the findings of recent research that i-vectors and PLDA-based solutions continue to be the dominant speaker recognition approaches for text-independent tasks .The fact that the results are significantly better when English was used both as a training and test language suggests that the solutions considered were optimized for the use of English. If the tested open-source software is used with another language, the optimal values of the configuration parameters in the software package must be carefully examined before it is used. This implies that the performance of speaker recognition methods varies by language.The analysis of the results shows that the values of the EER parameter in the first approximation are inversely proportional to the length of the sentences. Additional research should indicate the optimum sentence length at which the minimum possible EER is approached.Tested software solutions are very sensitive to the characteristics of the voice recorder, which in practice can significantly affect the performance of the speaker recognition or verification system. It is desirable to use a solution that, in preprocessing, minimizes the impact of variations in the characteristics of the speech recording device. Additionally, the results presented in this paper indicate potential problems that may occur when localizing a chatbot application requiring speaker verification.In further research, it would be interesting to examine the performance of speaker recognition systems which use voice as one of the biometric modalities, in two-factor authentication (2FA) or multifactor authentication (MFA).


SiC wafers, due to their hardness and brittleness, suffer from a low feed rate and a high failure rate during the dicing process. In this study, a novel dual laser beam asynchronous dicing method (DBAD) is proposed to improve the cutting quality of SiC wafers, where a pulsed laser is firstly used to introduce several layers of micro-cracks inside the wafer, along the designed dicing line, then a continuous wave (CW) laser is used to generate thermal stress around cracks, and, finally, the wafer is separated. A finite-element (FE) model was applied to analyze the behavior of CW laser heating and the evolution of the thermal stress field. Through experiments, SiC samples, with a thickness of 200 μm, were cut and analyzed, and the effect of the changing of continuous laser power on the DBAD system was also studied. According to the simulation and experiment results, the effectiveness of the DBAD method is certified. There is no more edge breakage because of the absence of the mechanical breaking process compared with traditional stealth dicing. The novel method can be adapted to the cutting of hard-brittle materials. Specifically for materials thinner than 200 μm, the breaking process in the traditional SiC dicing process can be omitted. It is indicated that the dual laser beam asynchronous dicing method has a great engineering potential for future SiC wafer dicing applications. SiC power devices have continuously increased their share in the high-power semiconductor market in the last decade and are used in a series of applications such as electric vehicles and urban rail transit. However, due to their hardness and brittleness characteristics, there is one bottleneck in the SiC device manufacturing field, which is the wafer dicing process. Currently, SiC wafers are mainly mechanically diced by diamond-coated blades with low feed rates, in the range of 5–10 mm/s, and a high risk of side chipping at the edges of the diced chips. Furthermore, the diameter of the 4H-SiC wafer has increased from 25 to 100 mm, and the 150 mm transition is upcoming. With the recent smaller and thinner trend in semiconductor manufacturing, mechanical sawing has reached its limit in SiC wafer dicing.To improve the dicing quality of SiC wafers, many novel dicing technologies have been developed to fulfill the requirements of throughput, edge quality, and costs, such as laser ablation cutting, plasma cutting, high-pressure water cutting, electrical discharge wire cutting, and water-jet guided laser cutting. The thermal separation method is a critical technology that is suitable for the dicing of hard-brittle materials . TypicalCurrently, there are two major processes based on the thermal separation method: non-premade trajectory cutting (NPTC)  and prem2 laser + water-cooled method for the thermal fracture cutting of glass. It was shown that increasing laser heat source power or applying cooling measures could increase the stress intensity factor at the initial crack to reach the threshold value quickly, to increase the cutting speed. Abramov et al. [NPTC is mainly used for the rough machining of thicker glass and ceramic plates and other thick materials, with no chips and microcracks in the middle of the cut trajectory. The objectives of the researchers are to increase the cutting speed, reduce the trajectory deviation, and improve the surface cutting quality. A practical method based on the principle of the NPTC to increase the cutting speed is to change the shape of the heat source energy distribution and application of cooling. Yamamoto et al.  used thev et al.  of Corniv et al. .The major problem in NTPC is trajectory deviation. Salman et al.  used a dThe surface quality obtained by the thermal cracking method effectively improves bending strength. Kondratenko et al.  investig2 laser in the cutting system of LCD glass substrates. The results showed that the introduction of ultrasonic vibration could improve the cutting speed greatly .Premade trajectory cutting is mainly used for the processing of liquid crystal display (LCD), plasma display (PDP), and flat panel display (FPD). PTC is characterized by faster cutting speeds and higher accuracy of the cutting trajectory. Many researchers have expanded the applicability of the PTC method for material dicing. Kang et al.  performe2 thermal laser for cracking to separate LCD glass. The experimental results showed that at low numbers of femtosecond laser pulses, the glass damage was small and the groove depth was not deep enough. When the number of femtosecond laser pulses was increased to six, the following CO2 laser was more effective in separating the glass by thermal stress. Wang et al. [2 laser and cooled by Ar gas.Cross-sectional quality is significantly improved by using PTC. KIM et al.  used a fg et al.  conducteStealth dicing ,18,19 isIn this work, we investigate a dual laser beam asynchronous dicing method by combining stealth dicing and premade trajectory cutting. The laser-based cutting method proposed in this work is clean, fast, and efficient and does not involve any chemical agent or liquid. The dependence of the laser process parameters on the cutting quality was theoretical and experimentally investigated. The quality of the cut edge was thoroughly analyzed by optical microscopy.The operation of the DBAD method in our experiments is as follows: firstly, stealth dicing is operated on SiC. The pulse duration of the laser used in this process is 750 fs, and the scanning speed is 3000 μm/s. Through the fine control of focus depth and single pulse energy, the surface has no visible cracks after the process. Then, a CW laser sweeps through the trace of SD, and the internal cracks from the stealth dicing are extended vertically due to the thermal stress. The parameters of the two lasers are listed in Briefly, 4-inch diameter silicon carbide (4H-SiC) wafers of 200 μm thickness were selected in this study; the physical properties of the material are shown in To better understand the heat accumulation process and stress concentration process caused by the continuous laser, a two-dimensional finite element model (FEM) was established. The thermal stress produced by the moving CW laser around the interior hole produced by the previous pulsed laser is shown schematically in p are the density and constant pressure heat capacity of the material, respectively. The formula of thermal stress is given by:0 is the original length of the material before the expansion. These three equations are coupled with each other and are resolved using COMSOL Multiphysics.High-intensity lasers, incident upon a material that is partially transparent, will deposit power into the material itself. The absorption of the incident light can be described by the Beer–Lambert law as it can be written in differential form for the light intensity I as:The component structure with mesh is shown in In this study, a series of experiments were conducted to cut SiC using a dual laser beam asynchronous dicing method based on the finite element simulation. Based on the principle of DBAD, single-pass stealth dicing is processed on the material, and then the thermal stress is generated using the 8 W-1040 nm continuous laser to extend the crack. Finally, the material is cut completely through the simple wafer expanding process. If the stealth cutting operation and thermal cracking are performed simultaneously, it will lead to misalignment and defects inside hard-brittle materials such as SiC, resulting in large errors in the positioning accuracy of the subsequent process.The experimental results are in agreement with the above FEM simulation results, proving the correctness of the FEM simulation model and the effectiveness of the DBAD method. The quality of processing is perfectly expected. Compared with traditional stealth dicing, there was no serious edge breakage and no wrong crack propagation occurred from the mechanical breaking process. During the stealth dicing operation, there was no remarkable crack trace on the surface, so that the processing window was expanded with an improved production rate. In addition, the following wafer-expanding process maybe not be necessary, especially for the processing of thin hard-brittle materials. It is also suitable for the processing of hard-brittle materials with a thickness less than 200 μm.In our experiments, the effect of the changing continuous laser power on the DBAD system was also studied. The thermal press is generated using a 1040 nm continuous laser. However, the crack trajectory may be bent with the power of the laser rising to 10 W compared with the crack using an 8 W continuous laser, as seen in Miniaturization and performance enhancement drives the development of wafer separation technologies. Especially for hard-brittle materials, chippings occur with a poor cutting quality using mechanical dicing, while traditional stealth dicing with a mechanical breaking process leads to edge breakage. It is shown that the validity of the DBAD method is certified because of the above finite element simulation and experiments. The processing flow for cutting hard-brittle materials is simplified by improving processing quality compared with traditional processing. An effective method is provided, using dual laser beam asynchronous dicing to cut hard-brittle materials such as SiC.In this paper, a dual laser beam asynchronous separation method for SiC wafer dicing has been put forward. In this method, a series of micro-cracks is firstly formed inside the wafer through a stealth dicing process by controlling the focal depth; the SD process will not induce visible cracks on the surface. Then, a CW laser is loaded on the dicing street, and the thermal stress leads to the wafer separation process.The absorption of the CW laser and the resulting thermal stress was calculated using a finite element model. The simulation indicated that the tensile stress produced by the CW laser heating in the upper and lower ends of the voids is the main mechanism of vertical crack propagation. To get better separation quality, the moving speed and the power of CW laser should be properly adjusted. Insufficient laser power will weaken thermal accumulation, and the SiC wafer will not reach its fractural strength. In contrast, excess laser power can cause unnecessary heat accumulation and potentially harm the circuit. Based on the simulation analysis, an experimental machine was built, and several 4H-SiC wafers with a thickness of 200 μm were cut; a neat cutting side wall without chipping was obtained. Through the experimental study, it can be determined that the proper depth of the last SD layer can be 50 μm below the surface, and the optimal moving speed of the CW laser is 1000 mm/s; acting with 8 W of laser power, the fracture will propagate upward stably, and the SiC wafer can be separated along the expected SD path.This novel DBAD method provides an effective solution for wafer cutting, specifically for hard-brittle materials with a thickness less than 200 μm, compared with other traditional processing methods.