Since deformation estimation may lead to errors occurring when the camera vibrates, it is necessary to remove the image global motion before computing real bridge deformation. In this study, a combination of image motion correction algorithm and 2D image-based deformation measurement technique was utilized to address the issue of camera motion during the image data acquisition for bridge deformation measurement. Based on the proposed methodology, the image motion parameters were estimated by defining an effective sub-image in the image and using Iterative Affine Motion Estimator. Then the estimated parameters were applied to all pixels of each captured image to remove the motion effect. Finally, the corrected images were used to analyze by a 2D image-based deformation measurement technique in order to extract and measure real bridge deformation by tracking artificial or natural targets. The proposed methodology was validated by two experiments in the lab and field environments. Achieved results show the accuracy and reliability of the proposed methodology. Deformation measurement is a key factor for safety and serviceability evaluation of bridge structures during their service life . To assuThe contact-based measurement methods can be implemented through conventional measurement devices including dial gauges, Linear Variable Differential Transformers (LVDTs), and accelerometers ,3,4. DiaNon-contact-based measurement approaches use high technology instruments in order to overcome the drawbacks of the conventional deformation sensors. Laser Doppler Vibrometer (LDV) ,6, Globa(1)Remote deformation measurement of bridges within the range of a few meters to hundreds of meters.(2)Fully automated with performance in a real time manner.Among all the aforementioned non-contact measurement methods, the image-based method may be more applicable for the following reasons:Non-contact image-based measurement methods have been widely used recently for deformation measurement of road infrastructures, especially for bridge deformation measurements. Based on the study conducted by Lee and Shinozuka  image prThe research presented thus far indicates that non-contact image-based techniques are superior to other methods; however existing techniques do bear some limitations. As can be seen in most of the image-based deformation measurement methods, a fixed-position camera was utilized to measure the bridge deformation. In the field, camera motion due to ambient conditions such as wind results in motion noise which affects measurements, leading to significant error in the deformation measurements. In some field measurement cases, it was suggested that the probable camera movement may be somewhat decreased by installing windshield equipment on the camera at the bridge site .It is also noted that some authors have reported the use of reference points for estimating camera motion relative to a movable camera. For example, Olaszek  developeAs such, the current study is devoted to the development, implementation and presentation of a non-contact image-based method for the measurement of bridge deformation considering camera motion. The proposed method deals with ambient camera motion without requiring any additional optical equipment or extra time and cost for arranging the measurement procedure. By the proposed method, the image global motion caused by the camera motion is estimated and the image is corrected before calculation of real bridge deformation. Therefore, first the image motion correction method is presented. Then the 2D image-based deformation measurement method is utilized to measure the real bridge deformation by tracking artificial or natural targets. Thirdly, the proposed methodology is validated through asphalt Uniaxial Compression (UC) test in the lab and finally the validated methodology is tested and verified on a real bridge in the field.In the remainder of this work, the principles of the algorithm for image motion correction as well as the 2D image-based deformation measurement technique are described. Then, the experimental study in the lab and the field is conducted to verify the proposed methodology. Finally, the future research challenges are presented.In this section, the methodology of deformation measurement under described conditions is presented. The proposed approach addresses two main challenges, camera motion due to ambient conditions such as wind in the field environment as well as accurate measurement of bridge deformation through a non-contact image-based method. The proposed methodology is a combination of image global motion correction algorithm and 2D image-based deformation measurement technique. This combination is generally called “2D image-based motion correction methodology” in this paper. In this section, an effective technique for the correction of the image global motion is described. Several research works have been conducted for image global motion estimation and correction ,39,40,41In this algorithm, the first image is considered as the reference image and the image sequence are continuously (frame by frame) analyzed to estimate the global motion parameters and remove the effect of camera motion. So, by following the IAME algorithm, the image global motion parameters, including translation, rotation, expansion, zooming, skewing and scaling are estimated. It means that the proposed image motion correction algorithm is capable of estimating and removing the camera translation and rotation in different camera axes. The final estimated global motion parameters obtained through analysis of ROI are then applied on the all image pixels; subsequently each image is corrected considering the image global motion.With all the captured images globally corrected, they were cropped from a fixed reference point at the bottom left of each image, thus preparing all the images for deformation measurement analysis. This fixed reference point could be selected by determining a crop point from among the existing elements in the image.After all the obtained image data were globally corrected, the 2D image-based deformation measurement technique was applied to calculate the real bridge deformation. The proposed image-based technique is a non-contact target-based method which is described below.First the intrinsic and extrinsic camera parameters are calculated through MATLAB Camera Calibration Toolbox . These pIn the next stage, the distorted image is converted into an undistorted image by applying the camera calibration results . With thThen an image processing and analysis algorithm is utilized to detect the target points including artificial or natural targets and estimate the center position of targets. It should be noted that in some situations, access to the monitored bridge and installation of artificial targets is difficult. So, the natural targets are installed and tracked through the image processing and analysis algorithm to reduce the required cost and time of bridge deformation measurement. The detailed flowchart of image processing and analysis algorithm is shown in As shown in Accordingly, all the successive images are considered to be analyzed for estimation of momentary metric position of target center at time he image .Finally, the wavelet transform is utilized to reduce the unexpected noise or variations of the obtained results . By applIn this section, the proposed 2D image-based motion correction methodology is tested in both lab and field environments to verify the capability of this method for addressing the issue of camera motion during image data gathering. First, asphalt Uniaxial Compression (UC) test is performed in the lab to validate the proposed methodology. In this experiment, a cylindrical asphalt specimen is loaded by the Universal Testing Machine (UTM) and deformation is measured and computed by both traditional and 2D image-based motion correction methods. Then the proposed methodology is applied on the real bridge in the field environment to show the reliability of the proposed methodology.In order to validate the proposed methodology in the laboratory, a test was performed on the asphalt specimen. This Uniaxial Compression (UC) test had been carried out by Sun et al. in 2016 . In thisIn this experiment, a cylindrical asphalt specimen with a height of 15 cm and a diameter of 10 cm was loaded under the variable vertical load, which resulted in vertical deformation in the specimen during the time of loading. This actual deformation of the specimen was measured using the LVDT installed at the top of the specimen. The time and applied load were also measured by the Universal Testing Machine (UTM) during the test. In addition, to estimate the vertical deformation through the 2D image-based motion correction methodology, four circular black targets with a 4 mm diameter The test image data and the real data from LVDT were gathered during the loading operation. The calibration image data was also collected before starting the main test. Also all image data captured by the camera is transferred to and stored on a hard drive.However, in the research work conducted by Sun et al. , it was Three types of image data were generated: original image data obtained through fixed camera (Group 1), image data generated by synthetically image translation and rotation or before applying the image motion correction algorithm (Group 2), and corrected image data after applying the image motion correction algorithm (Group 3).As for the data in the first group, it is possible to directly use the 2D image-based deformation measurement technique to calculate the asphalt vertical deformation. So, firstly, the original captured image data were undistorted considering the camera calibration parameters as mentioned in Then the images were processed and analyzed in order to detect the target and determine the pixel position of each target center during the time. The circular target detected by applying the image processing and analysis algorithm is shown in Having the pixel locations of the target center in each image sequence, as well as the image scale factor, the asphalt vertical deformations were computed. Subsequently the metric positions of each target center during the time were estimated using Equation (4). It should be noted that in this experiment, it is assumed that the diameter of a circular target is considered as the known physical length on the asphalt surface in order to estimate the scale factor. Finally, the wavelet transform was used to reduce the existing noise of the results.For the analysis of the image data in Group 2, the same method as the first group was applied to calculate the asphalt vertical deformation. Since the images were translated and rotated in this state; thus the achieved deformation results were not be going to be reliable. If the camera vibrated due to ambient conditions, direct use of the 2D image-based deformation measurement technique would not be effective to achieve reliable results. Therefore, in order to solve the problem, first it is necessary to correct the image data through the image global motion correction algorithm as described in the previous section and then apply the 2D image-based deformation measurement technique to estimate the vertical deformation.As such, the image motion correction parameters were first estimated by taking into account a sub-image called ROI of the main image. Subsequently the obtained parameters were applied on all pixels of the entire image, as shown in Finally, by applying the proposed methodology, three categories of results are achieved from three groups of images, including original images, synthetically moved images (before image motion correction) and corrected images (after image motion correction). Then, by comparing the achieved deformation results from analysis of all image data groups to each other and with those obtained through LVDT, the proposed methodology, including the image global motion correction algorithm and the 2D image-based deformation measurement technique were validated. It should be noted that the deformation results of LVDT are only available at the location of target Based on the proposed methodology, the image data  were analyzed and the comparison was accomplished by comparing the deformation results of asphalt specimen through analysis of the three groups of image data previously described. First, these results are compared at target (a)deformation measurement using Group 2, Group 3, and LVDT,(b)deformation measurement using Group 1, Group 2, and Group 3,(c)deformation measurement using Group 1, Group 2, and LVDT.In The comparison results showed that the deformation results obtained from the analysis of corrected images (after image motion correction) agree with those obtained from the analysis of original images or LVDT as the ground-truth value a,b. ThisThe existing errors were calculated and summarized in After validating the proposed methodology by deformation measurement analysis at target According to After verification of the proposed methodology in the lab, the real bridge experiment in the field environment under the vibration condition was conducted in order to measure the dynamic deformation of bridge.In this section, a real bridge experiment in the field environment was conducted to investigate the reliability and efficiency of the proposed methodology. The test was performed at a new highway concrete bridge located in a mountainous area in Yunnan Province, China, with a total length of 336 m. It has three spans including one middle (length: 160 m) and two lateral spans (length: 88 m). Two separate parallel congruent unidirectional bridges span the valley to manage traffic flow coming from the opposing mountain. The bridge structure and its specifications are shown in As can be seen in Since the camera was placed on the bridge deck, some ambient conditions such as bridge vibration caused by traffic flow or wind may affect the fixed position of the camera and subsequently, the camera may vibrate or move during image data gathering.For bridge loading, a heavy cargo truck was passed over the bridge to apply the standard dynamic load. At the same time, the image data gathering was also performed during the bridge loading. Since the camera vibrated during the image collection procedure, the captured images were subjected to motion. In this case, the proposed 2D image-based motion correction methodology which was previously developed and verified was utilized to estimate and measure bridge vertical dynamic deformation considering image global motion.Therefore, the global motion of images was corrected using the image global motion correction algorithm described. The affine motion estimator was applied to the defined ROI in order to estimate the motion correction parameters. Subsequently, the whole image was corrected by applying these parameters to all pixels of the image as shown in In addition, a natural point near target A was defined to be detected and tracked as shown in The captured images were corrected by analyzing the images obtained through the vibrated camera. Subsequently, the bridge deformation was calculated under dynamic load. In this experiment, two groups of images, those before and those after image global motion correction were analyzed. As indicated in Considering the experiments conducted in this study, it was proven that the real deformation of a bridge can be accurately calculated in the presence of camera motion. Compared to previous researches regarding bridge deformation measurement, it was found that the proposed methodology is a convenient, efficient and cost effective method for estimating bridge deformation in the presence of camera motion. The proposed methodology can be utilized as a primary step for addressing the problem of deformation measurement considering the camera motion.-According to the proposed method in this study, fixed background features are required to be defined for estimation of camera motion parameters. But finding stationary background features is challenging in some situations, for example on the shoreline. So, this challenge can be addressed by using inertial measurement unit (IMU) sensors in the Unmanned Aerial Vehicle (UAV) for camera motion estimation that can be investigated in the future.-Different issues may affect the accuracy of bridge deformation measurement including: the distance between camera and bridge, using various types of cameras, and using different lenses. These issues can also be considered as the future researches.-In this study, 2D image-based deformation measurement technique was developed to measure the in-plane vertical bridge deformation in presence of camera motion as described in While reliable results were achieved by the proposed method, some research issues can be investigated in the future which are explained in the following: The camera motion due to ambient conditions such as wind is a problem for accurate bridge deformation measurement, especially in the field environment. It is necessary, therefore, to find and develop an efficient image-based methodology to use the moved images as input and compute the real bridge deformation accurately and conveniently. This paper focused on the development and implementation of an image-based technique by which the issue of camera motion is addressed in less time with less cost. In addition, a combination of image motion correction algorithm and 2D image-based deformation measurement technique was developed to measure and extract real bridge deformation from the vibrated images in a continuous and convenient manner. Finally, two experiments were conducted (one in the lab and the other in the field) to test and verify the proposed methodology. The obtained images were categorized in different groups , analyzed based on the proposed methodology through MATLAB. Then the real deformation was estimated for each image data group and compared. In addition, the validity of the proposed methodology was investigated by detecting and tracking artificial and natural targets. Comparing the deformation results obtained through analysis of image data before and after motion correction with those obtained through original images and LVDT verified the accuracy and efficiency of the proposed methodology. This methodology can be considered a suitable method for addressing the issue of camera motion.Given the challenges addressed in this study through the proposed methodology including detection of fixed natural background features within the image to calculate camera motion parameters, removing camera motion (translation and rotation) by developed image motion correction algorithm, estimating the bridge displacement using one camera and 2D image-based deformation measurement technique, and measuring the vertical in-plane deformation of bridge structure, a monitoring base was made toward measuring the bridge deformation in presence of camera motion. In addition, the challenges which can be investigated in the future were presented, including the usage of UAV for measuring bridge displacement, adopting IMU sensors to estimate camera motion, and using multiple cameras for measuring out-of-plane motion of the structure. Considering the contributions of this study regarding the bridge deformation measurement in presence of camera motion, and the described possible future studies, it is expected that the capabilities of the proposed methodology can be extended and improved in the near future.


In this paper, the development of an eye-tracking-based human–computer interface for real-time applications is presented. To identify the most appropriate pupil detection algorithm for the proposed interface, we analyzed the performance of eight algorithms, six of which we developed based on the most representative pupil center detection techniques. The accuracy of each algorithm was evaluated for different eye images from four representative databases and for video eye images using a new testing protocol for a scene image. For all video recordings, we determined the detection rate within a circular target 50-pixel area placed in different positions in the scene image, cursor controllability and stability on the user screen, and running time. The experimental results for a set of 30 subjects show a detection rate over 84% at 50 pixels for all proposed algorithms, and the best result (91.39%) was obtained with the circular Hough transform approach. Finally, this algorithm was implemented in the proposed interface to develop an eye typing application based on a virtual keyboard. The mean typing speed of the subjects who tested the system was higher than 20 characters per minute. The interest in eye detection applications has been considerably increasing . Many eyEye tracking is the process of measuring either the point of gaze or the motion of an eye relative to the head. An eye tracker is a device for measuring eye position and eye movement ,13,14. TDifferent types of eye tracking systems have been reported in the literature ,21,22,23The increase in computing power has led to diversification of pupil detection algorithms (PDAs). The pupil detection algorithms presented in the literature can be classified into feature-based and model-based approaches ,29.Feature-based approaches involve detecting and localizing image features related to the position of the eye pupil. The best shape approximation of the eye pupil image provided by an infrared (IR) video camera is an ellipse. An ellipse can then be fit to the feature points using several techniques: least-squares fitting of ellipse (LSFE) ,31; votiThe LSFE algorithm performs the fitting of a conic (ellipse) to a set of data points by minimizing the sum of square algebraic distances of the feature points to the conic that is represented by the ellipse coefficients.The CHT algorithm uses a parameter accumulator. Every edge pixel in the eye image votes for the parameters of all circles in which it can be a part. Maxima in the accumulator correspond to discovered circles . Since aThe RANSAC algorithm is an iterative procedure that selects many small but random subsets of the input data, uses each subset to fit a model, and finds the model that best fits the input set of data points ,35. The Model-based approaches do not explicitly detect features but rather find the best fitting model, which can be represented by either a circle or an ellipse, that is consistent with the eye image . In the Most studies presented in literature focused on the performance of different types of pupil detection algorithms on static eye images. Fuhl et al.  conducteOne of the most recent and accurate pupil detection algorithm in pervasive scenarios, Pupil Reconstructor (PuRe) , is baseMost of these algorithms are based on the principles mentioned above; the PDA proposed by Świrski et al.  is basedIn none of these studies was the real-time performance of PDAs in a scene image analyzed, many reporting a detection rate obtained only for raw eye images provided by the video camera. These studies focused on determining the algorithm accuracy when analyzing static eye images with low quality acquired in difficult conditions . Other algorithms reported in the literature provide a detection rate or an algorithm running time that are inadequate for real-time applications, as is the case for the algorithm based on new projection function .To be used for eye tracking applications, the real-time performance of the PDAs must be investigated too. The experimental results illustrated in the literature for static eye images show that the accuracy of each pupil detection algorithm is strongly dependent on the characteristics of the eye tracking system used for acquiring these images.In this paper, the development of an eye-tracking-based human–computer interface for real-time applications is presented. To find the best software solution for the proposed interface, we comparatively analyzed eight pupil detection algorithms based on the most representative state-of-the-art techniques for both static eye images and real-time applications using a novel testing protocol for evaluating the eye tracking system performance in the scene image. Of the eight PDAs analyzed in this study, we created six: three were developed in previous work in the field of eye detection, based on the CHT algorithm , LSFE meThe real-time performance of each algorithm was evaluated using the head-mounted eye tracking interface developed by our team on a set of 30 subjects in different lighting conditions, resulting 240 distinct use cases, for a total of 370,182 eye images processed by the system.This paper shows that the performance of any eye tracking system for real-time applications depends not only on the algorithm accuracy for raw eye images, but also on other factors, such as cursor controllability and stability on the user screen, running time, lighting conditions, the type of eye tracker device, and the user ability in using the system. According to these considerations, the algorithm based on circular Hough transform is shown to be best for real-time applications developed on the proposed eye tracking interface. As an application of this system, a low-cost eye typing application using a virtual keyboard is also presented.The rest of this paper is organized as follows: Video oculography is a technique based on video analysis of eyeball movement using a video camera in the visible or infrared (IR) spectrum. The advantages of the methods based on the analysis of infrared eye images provided by video cameras include their versatility and being practically independent of lighting conditions and individual characteristics of the eye. The main disadvantage of infrared eye image capturing is that this technique cannot be used outdoors during daytime due to the ambient infrared illumination ,29.Infrared eye tracking applies either the bright-pupil technique (the pupil is clearly demarcated as a bright region due to the photo reflective nature of the back of the eye ) or darkThe head-mounted eye tracking interface used in this study included an infrared video camera ) with a resolution of 640 × 480 pixels (px) mounted on frame glasses immediately underneath the eye , connectTo integrate gaze detection techniques into human–computer interfaces for real-time applications, it is necessary to implement widely available, reliable, high speed, and high accuracy pupil detection algorithms compatible with the hardware used by present-day computers.The proposed eye tracking interface moves a cursor on the user screen in real-time according to the user gaze direction. To implement different real-time applications, the user screen is divided into many cells. Target objects are placed in each cell that can be selected by the user’s gaze direction, according to their will or need. A target object selection is performed by focusing the user gaze direction in its selection area for a certain dwell time, established according to the user ability in using the system. The dwell time must be chosen to avoid the Midas touch problem , which iThe compound eye tracking algorithm used for real-time applications implemented in our eye tracking system includes the following main procedures, described by Bozomitu et al.  in detaii, i = 1, 2, …, 9; 1 placed in the center of the user screen.The calibration stage is used to determine the mapping function coefficients and the radius of the eye pupil, required for optimal operation of the quantitative binarization procedure. The coefficients of the mapping function between the raw eye image and the scene image are obtained during the calibration process when a set of targets in known position  to determine the most appropriate algorithm for real-time application.The accuracy of the eye tracking system strongly depends on the precision of the calibration stage, during which the coefficients of the mapping function are determined. The convex shape of the eye causes nonlinear eye movement. Thus, the mapping function between the raw eye image provided by the IR video camera and the scene image is a user-dependent nonlinear function. Nonlinear mapping functions between the raw eye image and the scene image have the advantage of smooth changes in mapping across the user screen. The most common nonlinear function is biquadratic, introduced by Sheena and Borah in 1981 . BozomitTo evaluate the PDA accuracy for real-time eye tracking applications, the mapping rate (MR) between the raw eye image and the scene image must be known. The MR depends on the calibration stage accuracy, the type and order of the mapping function used, the resolution of the user screen, and the user’s eye physiology. The mapping rate is defined as the radius of the circle resulting on the user screen when a circle with radius of 1 pixel is mapped between the raw eye image (with a resolution of 640 × 480 pixels) and the scene image (with full high definition (HD) resolution).Because the performance of any eye tracking interface depends on the calibration procedure quality, any eye tracking system must use an indicator of the accuracy of the calibration stage. This indicator can be determined by evaluating the system mapping rate during the calibration stage. Due to the nonlinearity of the mapping function, the mapping rate parameter must be calculated in different positions on the user screen. The best positions used to calculate the mapping rate are indicated by the nine identical quadrants on the user screen illustrated in r between 1 to 10 pixels are mapped between the raw eye image and the scene image, resulting in an irregular geometric shape that can be approximated by an ellipse or a circle. The radii of these circles in the raw eye image represent the algorithm accuracy in term of pixel error for any possible direction of the detection error.To calculate the MR of the proposed system, circles with radii iR(r),  depending on mapping circle radius r. The mean value of the mapping rate for each quadrant in the scene image, iMR, illustrated in The shapes resulting on the user screen are not perfectly circular due to the nonlinearity of the mapping function used and an inaccurate calibration process. For simplicity, in the following analysis, these shapes are approximated by circles with radii Thus, the mapping rate of the proposed eye tracking system can be approximated as the mean value of mapping rates for each quadrant, as follows:R(r) of the circle that approximate the shape resulted on the user screen when a circle with radius r is mapped between the raw eye image and the scene image can be approximated by the following linear function:The radius The MR is calculated for each subject during calibration stage and represents an important indicator of the calibration stage accuracy. The experimental results for 30 subjects in laboratory conditions showed that the MR for the propose system varies between 8.86  to 15.96 . Values higher than 16 for this parameter resulted in a decrease in the system accuracy for real-time applications and are not allowed during the calibration stage. For these situations, the calibration stage was resumed until a value less than 16 was obtained for MR.MR = 9, a circular target area with a radius of 45 pixels can be precisely controlled on the screen by the user’s gaze direction. This size of the target object is in agreement with the keypad dimensions of the virtual keyboard developed by OptiKey 27].At the end of the test session, the participants completed the System Usability Scale (SUS) questionnaire to determine the performance of the eye tracking interface based on CHT algorithm. Each item’s score contribution ranges from zero to four (with four being the most positive response). The SUS score is calculated according to the procedure previously illustrated .(S1) I think that I would like to use this eye tracking interface frequently.(S2) I found the eye tracking interface unnecessarily complex.(S3) I thought the eye tracking interface was easy to use.(S4) I think that I would need the support of a technical person to be able to use this eye tracking interface.(S5) I found the various functions in this eye tracking interface were well integrated.(S6) I thought there was too much inconsistency in this eye tracking interface.(S7) I would imagine that most people would learn to use this eye tracking interface very quickly.(S8) I found the eye tracking interface cumbersome to use.(S9) I felt very confident using the eye tracking interface.(S10) I needed to learn a lot of things before I could get going with this eye tracking interface.The 10 statements on the SUS are as follows :(S1) I tTo improve this situation, the subjects received a practice phrase before starting the experiment. The average value of 89.50 indicates the good performance of the eye tracking system according to the scoring system established in specialty literature .In this study, we developed an eye tracking-based human–computer interface for real-time applications. The most suitable algorithm for the eye tracking interface was determined by analyzing the performance of different PDAs on static and video eye images.To evaluate the algorithm accuracy for video eye images, we proposed a new method to determine the mapping rate between the raw eye image and the scene image. Using this method and knowing the algorithm accuracy for static eye images, we determined the algorithm accuracy expected for real-time applications as the minimum size of the target area on the user screen, which can be precisely controlled by the user’s gaze direction. We showed that the high precision of a PDA on static eye images does not guarantee high performance for video eye images. To ensure high performance for real-time applications, other parameters, like cursor controllability and stability on the user screen and algorithm running time, were also considered.Experimental sessions involved testing the eye tracking interface that uses all eight PDAs on a set of 30 subjects in laboratory conditions using a new testing protocol developed using scene images for different real-time operating scenarios and lighting conditions. The resulting 240 use cases were statistically processed.The experimental results obtained by the proposed PDAs  produced detection rates between 84.46% and 91.39% in the scene image for a target area radius of 50 pixels on the user screen. The best results in term of accuracy and cursor controllability and stability on the user screen were achieved using the CHT-based algorithm, which was the most user-friendly for the developed interface.The performance of the proposed low-cost eye tracking interface based on the CHT algorithm was evaluated on an eye typing application using an on-screen virtual keyboard. The mean typing speed obtained from a set of 30 subjects was 20.74 characters/min with a mean total error rate of 3.55%.Considering the low price, high accuracy for variable and non-uniform lighting conditions, and the results of the SUS questionnaire, the proposed eye tracking interface can be a valuable solution for different types of real-time applications. Depending on the application type and the device used, a trade-off between accuracy, controllability, running time, noise sensitivity, and performance of the PDA in nonuniform lighting conditions must be considered in any eye tracking system design.


Neurally inspired robotics already has a long history that includes reactive systems emulating reflexes, neural oscillators to generate movement patterns, and neural networks as trainable filters for high-dimensional sensory information. Neural inspiration has been less successful at the level of cognition. Decision-making, planning, building and using memories, for instance, are more often addressed in terms of computational algorithms than through neural process models. To move neural process models beyond reactive behavior toward cognition, the capacity to autonomously generate sequences of processing steps is critical. We review a potential solution to this problem that is based on strongly recurrent neural networks described as neural dynamic systems. Their stable states perform elementary motor or cognitive functions while coupled to sensory inputs. The state of the neural dynamics transitions to a new motor or cognitive function when a previously stable neural state becomes unstable. Only when a neural robotic system is capable of acting autonomously does it become a useful to a human user. We demonstrate how a neural dynamic architecture that supports autonomous sequence generation can engage in such interaction. A human user presents colored objects to the robot in a particular order, thus defining a serial order of color concepts. The user then exposes the system to a visual scene that contains the colored objects in a new spatial arrangement. The robot autonomously builds a scene representation by sequentially bringing objects into the attentional foreground. Scene memory updates if the scene changes. The robot performs visual search and then reaches for the objects in the instructed serial order. In doing so, the robot generalizes across time and space, is capable of waiting when an element is missing, and updates its action plans online when the scene changes. The entire flow of behavior emerges from a time-continuous neural dynamics without any controlling or supervisory algorithm. Neurally inspired robotics already has a long history. To position our work in this history and review our conceptual commitments, we discuss three strands of neurallly inspired robotics.One strand goes back to Grey's electronic turtle Grey,  and Braiautonomously in the sense that sensory information from a structured environment may trigger the activation of elementary behaviors, which may lead to chains of activation and deactivation events through the architecture, inducing sequences of behavioral decisions, without the need for an explicit internal plan, schedule, or program. The organization of such behaviors is implicitly encoded in the architecture itself.By organizing closed action-perception loops in architectures, most famously the subsumption architecture Brooks, , this foAvoiding representation and abstraction is a feature of the approach Brooks, , but alsOur approach is historically based on behavior-based thinking, which we extended by adding neural memory representations and neural mechanism of decision making  data  It autonomously builds scene memory, a neural map of locations and feature values bound to those locations. Different objects are sequentially brought into the attentional foreground, in each case creating an entry into scene memory, which can be updated if change is detected. (2) The system learns the serial order of events that occur in its visual array. Each time an attended object changes, the system registers the transition and learns the new feature value as associated with its serial position. This provides a possible interface through which a human user can interact with the system. (3) The system generates a sequence of actions oriented at objects in the world in the learned serial order. At any point in the sequence, this entails finding an object in the visual surrounding that matches the feature values currently sought, generating the action, and then transitioning to the next sub-task within the learned sequence. This exemplifies the capacity of the system to autonomously generate organized behavior that is not merely reactive but reflective of a learned plan. (4) Each action consists of a pointing gesture oriented at an attended object. The action is initiated once the object has been brought into the attentional foreground, but may be updated any time if object shifts to a new location. This is a minimal instantiation of object-oriented action that any form of cognitive robotics must be capable of.Key to this demonstration is the notion of neural dynamics, in which strongly recurrent neural networks, approximated as spatially and temporally continuous neural fields, evolve primarily under the influence of their internal interaction that sets up attractor states. Inputs induce instabilities that bring about switches of neural states from which sequences of cognitive or motor states emerge. Such neural dynamics are capable of making decisions, building working memories, and organizing sequential transitions  . So even though neural populations may be driven by input, they may realize non-unique mappings from input to activation states based on their activation history.When different populations are coupled, they may induce these kinds of instabilities in each other. This is the basis for generating sequences of neural activation states. When the coupling occurs between excitatory and inhibitory sub-populations, the instabilities may trigger active transients, well-defined time courses of neural activation from temporally unstructured input. Neural oscillations are another possible dynamic regime.x. This leads to the notion of neural dynamic fields, u(x). We employ a particular mathematical formalization of the dynamics of such neural populations that goes back to Amari . Thus, one may think of the activation variable, u, as something like a population-level membrane potential that reflects how close neurons in the population are to the firing threshold [other formalizations use the firing rate as a population variable, see Wilson and Cowan , subject to a neural dynamics analogous to Equation (1).Fields may represent low-dimensional metric spaces. When their dimensionality grows, the binding problem arises and can be solved, see Chapter 5 of Schöner et al. . A limitWhen fields of different dimensionality are coupled, new functions emerge  the activation of the Saliency field, ∫ σ)dc the activation of the Working Memory projected onto the two spatial dimensions, x and y, and ωsel the field's selective lateral interaction kernel. Each input, σuin, to the field is weighted by a specific weight, win. The same notation is used in all following equations and the concrete parameter values can be found in the where Scene Space Selection, Working Memory, and WM Space Selection, ensuring that the color features represented in those fields originate from the same location. The Scene Space Selection field combines sub-threshold activation from the Space Color Maps field, that represents color information in the scene, with spatial sub-threshold activation from the Saliency Selection field. Together, these inputs induce a single peak in three dimensions that represents the attended spatial location and the color perceived at that location.The currently attended location achieves spatial feature binding: It is forwarded to the three 3D space-color fields, Saliency Selection field to add a peak in the 3D-Working Memory field.Working Memory field receives additional input directly from the camera image that includes the robot's arm. That input is proportional to the saturation channel from which the table color saturation has been subtracted. This mask, seen, for instance, in the Working Memory row of That color is extracted and combined with spatial information from the umem, of the Working Memory field is governed by the following dynamics:The activation, hmem describes the field's resting level, usel the activation from the Saliency Selection field, inbk the saturation channel from the camera image, ∫ σ)dxdy the activation of the Scene Space Selection field projected onto the color dimension, c, and ωmem the field's lateral interaction kernel.where Working Memory field is forwarded to the Memory Space Selection field, which works analogously to the Scene Space Selection field and thus forms a single 3D activation peak, representing color and spatial location of the attended location in working memory.Supra-threshold activation of the Scene and Memory Space Selection fields is forwarded to the Color Match field. That field forms a peak only when the input from the scene overlaps in location and color with one of the peaks in the memory field. A peak in the match field thus signals successful entry of an item into the Working Memory at the currently attended location. Supra-threshold activation in the match field projects onto the CoS Explore node, which in turn inhibits the Explore intention node. Deactivation of the Explore intention node removes the resting level boost from the Saliency Selection field inducing a reverse detection instability that propagates to the Scene and Memory Space Selection fields, the Color Match field and ultimately to the CoS Explore node. The newly created peak in the Working Memory field is sustained and the Explore intention node is released from inhibition enabling attentional selection of a new location.Color information represented in the Saliency field. Both layers, udfa and udsl, are two-dimensional fields over image space that are governed by the following dynamics with timescales, τdfa < τdsl:The scene representation sub-network is capable of detecting sudden object movement with the help of a two-layer offset detector connected to the hdet describes the common resting level, and σ) the thresholded activation of the Saliency field which excites the slower layer, udsl, and inhibits the faster layer, udfa. Because inhibitory input is stronger than excitatory input (wsin > wdsl), static visual structure induces supra-threshold activation in the slow layer, udsl, not in the fast layer, udsl.where wsin, vanishes faster than the excitatory influence from the slow layer, udsl, leading to the formation of a peak in udfa that represents the detection of an object that moves away from the location of the peak.Once an object is removed from the scene, the inhibitory influence, Reach intention node activates the Oscillate intention node, which initiates an active transient  are generated until the arm reaches the represented target.The sub-network responsible for reaching movements, based on Schöner et al. , autonomient see . The CosTarget Position and the EEF Position DNFs, respectively. Activation originating from working memory causes the creation of a peak in the Target Position field. Proprioceptive information from the current arm configuration is mapped through a forward kinematics into end-effector space and then transformed from rate to space code inducing a peak in a two-dimensional EEF Position field. Target and end-effector representations are cross-correlated with each other to create an end-effector centered representation of the target position. This representation is input into a two-layer field of neural oscillators, uexc and uinh. The faster excitatory layer, uexc, generates an active transient illustrated in uinh:Target and end-effector (EEF) are both represented as peaks of activation in two-dimensional fields defined over image space, the exc < τinh are the different relaxation times, hosc the resting level, σ) the end-effector-centered target representation, σ(uosc) the homogeneous resting level boost from the Oscillate intention node, and θ – a semi-linear threshold function.where τuexc), is transformed into a rate coded Cartesian velocity vector, v, using a set of feed-forward weights, wvel:The thresholded activation, θ, describe a linear distance function in the end-effector centered representation of the target position. For different movement distances, , these weights are tuned such that the arm reaches the target position within a fixed movement time. The velocity vector, v, is transformed into a joint velocity vector, J+, which depends on the current joint configuration λ(t):The weights, For more details on the generated velocity profile see Schöner et al. .EEF-Position field is gated by the Oscillate intention node. The EEF-Position field thus effectively represents the initial position of the hand. Termination of the transient is detected by the CoS Oscillate node, which receives excitatory activation from uinh and inhibitory activation from uexc. Activation of CoS Oscillate inhibits the Oscillate intention node, which resets the oscillator, and releases the EEF Position Gate from inhibition so that the end-effector position is updated. When the target representation overlaps sufficiently with the updated EEF Position, a peak forms in the Position Match field and activates the CoS Reach, which terminates the reach.While the oscillator is going, its input is not updated, because the connection from proprioception to the ordinal node and a feature field, here the one-dimensional Sequence Color field. Supra-threshold activation in a particular ordinal node thus induces a peak in the Sequence Color field that represents the color associated with that particular stage in the sequence.The serial order sub-network, based on Sandamirskaya and Schöner , allows The sub-network consisting of ordinal nodes, memory nodes and a single CoS node enforces the sequential activation of ordinal nodes in a fixed order:oi, representing the ith position in the sequence, inhibits all other ordinal nodes, oj, and activates its own self-sustained memory node, mi. The memory node pre-activates the next ordinal node, oi+1, through an excitatory connection and inhibits its own ordinal node, oi, to prevent it from becoming reactivated after completion of the stage. While activated, an ordinal node's self excitation, woi,oi, is sufficient to overcome inhibition from its memory node, wi,oim. An ordinal node remains active until the CoS node, uCoS, is activated and destabilizes all ordinal nodes, which, in turn, removes input from the CoS node that deactivates. The self-sustained memory nodes are unaffected, so that upon release from inhibition by the CoS, the pre-activated ordinal node of the next element in the sequence is activated. Recurring activation and deactivation of the CoS node thus creates a sequence of autonomous transitions between sequence elements in the order of ascending i. Ordinal and memory nodes can become activated only in the presence of an excitatory boost, w+h, from one of the task nodes, Learn (ulrn) or, Recall (urcl). Deactivation of an active task node leads to deactivation of all memory and ordinal nodes, effectively resetting the entire system.An active ordinal node, woi,ucol, between the active ordinal node, oi, and the active region in the Sequence Color field, ucol, are strengthened according to a dynamic version of the Hebbian learning rule:Connection weights, ulrn the activation of the Learn task node that gates the learning process.where η describes the learning rate and Sequence Color field arise when a color attended in the Working Memory Selection field is input through the gate field, Learn color, ulcol:Before learning, peaks in the woi,ucol(c), of an ordinal node, oi. The selective kernel, ωcol, ensures that only a single color is represented at all times.After learning, peaks in the Sequence color field may arise from previously learned connections, Learn and Recall, which alter the resting level of certain sub-sets of fields.The full network may operate in two different regimes: In the learning regime, a sequence of colors is presented to the system and learned. In the recall regime, a learned sequence of colors is reproduced by pointing at colored objects in a specific order. Each regime is evoked by the activation of its corresponding task node, Both task nodes boost the resting level of all ordinal and memory nodes to allow supra-threshold activation. When task nodes are deactivated, the removal of the corresponding boost causes activation of all self-sustained nodes to decay, effectively resetting the system. This happens, for instance, at the end of the sequence due to activation of the sequence's condition of satisfaction.Learn Color field, the Learn node enables that field to form supra-threshold peaks. At which color such a peak is erected is controlled by input from the Memory Space Selection field that represents the color at the currently attended location. That color is then imprinted in the connections to the currently active ordinal node through the learning dynamics (Equation 9). The Learn node pre-activates the Offset Detected node, which connects to the Sequence CoS. Thus, whenever a single object is presented in the learning regime, its color is associated with the currently active ordinal node and its removal from the scene causes a transition in which the active ordinal node is replaced by the next ordinal node.The Learn node acts as a gate between the Scene Representation and the Serial Order sub-networks. By boosting the Recall Color gating field so that the color represented in the Sequence Color field is passed on to the three-dimensional Memory Color Selection field. If an object in working memory overlaps with that color, a peak forms in the Memory Color Selection field. The peak's spatial position is forwarded to the Target Position field of the Arm Movement sub-network, which initiates a reaching movement. Once a reach has been successfully performed, the Reach CoS is activated, which triggers the Sequence CoS, causing the transition to the next ordinal node. In the recall regime, the arm will thus move autonomously to colored objects in the learned order, as long as appropriately colored objects are visible in the scene.The Recall node is a gate between the sequence generation and the arm movement sub-networks. It boosts the In this section we show how activation within the network unfolds in time during the learn and recall tasks. We visualize relevant activation fields to illustrate how the network's autonomy enables it to cope with variable timing during learning and with changes of the scene during recall.Cedar  still carries supra-threshold activation, exciting the fast layer leading to the formation of a peak, which will subsequently activate the Sequence CoS node inhibiting all ordinal nodes. This deactivates o1 and causes the color peak in the Sequence Color field to vanish as it is no longer supported by either learned connections nor color input from the scene. The missing input in the scene will also ultimately lead to a decay of activation in the slow Offset Detector layer and subsequently cause a reverse-detection instability in the fast layer and the Sequence CoS node.The object is removed from the scene at Sequence CoS node is followed by an activation of the next ordinal node o2 at t3. Between t2 and t3 a blue object has been added to the scene, whose color is then connected to the freshly activated ordinal node via the Hebbian learning rule. Removal of the object at t4 triggers the Offset Detector and the CoS node enabling the activation of the next ordinal node o3 at t5. The presented purple object is kept in the scene for a longer time span than the green or blue one, which does not influence the learning as the transition to the next sequence element at t6 is based on the removal event rather than timing.The deactivation of the Sequence Color, the Target Position, and the Position Match field at discrete points during the sequence recall.We demonstrate successful sequence recall through a pointing task, where the network moves the arm to an object in the scene matching the color of the current sequence element. Only a successful reach toward that object allows a progress to the next sequence element. An exemplary recall of three sequence elements is depicted in t0, the Recall task node has been activated, which lead to the activation of the first ordinal node and the emergence of a peak in the Sequence Color field at the green location due to the learned connections, wo1,ucol. The color information converges with the content of the Working Memory field in the Memory Color Selection field to form a three dimensional peak specifying position and color. Positional information is projected to the Target Position field of the Arm Movement sub-network, where it is forwarded to the movement generating oscillator and the Position Match field, which compares the current end effector position (center/left) with the current target position (bottom/right).At point t1, which is represented through a peak in the Position Match field that activates the Sequence CoS deactivating the current ordinal node. The CoS node itself falls below threshold activity as soon as the peak in the Position Match field destabilizes through a missing target representation that vanished through insufficient color input from the Sequence Color field.Due to a successful arm movement both positions match at point o2, which is associated with blue color. At t2 however the blue peak has emerged in the Sequence Color field, but the target position has not yet been extracted from working memory. The column of point t3 depicts the end of the movement, where the overlap of end effector and target cause a peak that triggers the Sequence CoS. In this particular configuration the match representation is only possible due to the self-sustaining working memory representation that shields the blue object representation from the occlusion through the arm.The missing inhibition from the CoS causes an activation of the next ordinal node t4 until t6 follows an analog activation pattern in which the ordinal node causes the formation of a purple peak in the Sequence Color field, which causes an extraction of the target position, leading to movement that terminates due to an represented match of positions. The movement times of all three movements are roughly the same despite their differences in distance, which results from the movement oscillator that enforces the same movement timing for all movements. See The movement toward the purple object depicted from Memory Color Selection field.The autonomy of all three parts of the field network makes the execution of the recall task robust against unforeseen changes in the scene. We demonstrate this in an exemplary recall episode, where one of the objects in the scene is moved while its color corresponds to the active sequence element. The episode is depicted in t0 in the Memory Color Selection field, where the green object forms a peak as it overlaps with the green color slice specified by the sequence color, while the purple object is present as a sub-threshold activation blob, and the blue object is entirely absent. As the first movement is finished at t1 all three objects are present in working memory as sub-threshold activation blobs.In this episode, build-up of the scene memory starts simultaneously with activation of the recall task, which causes a delay between the activation of the first ordinal node and the first movement as the green object, which is the first sequence element, is the second object committed to memory. This can be observed at t2, the second movement starts closely after the activation of the second ordinal node with the blue object as the target on the right side of the camera image. While the arm is moving the object is moved to the center/top position of the image, which results in a non-match between arm and target at the end of the movement, which can be seen at t3. Here working memory has updated the position of the blue object, which leads to an extraction of a different target position that does not match with the current position of the end effector. Only at t4 after a second movement was generated, the blue object and the end effector match, which concludes the recall of the second element of the sequence.Thus at t6.The last movement toward the purple object is then conducted without any further perturbations and terminates after a single movement at In this second recall episode demonstrating the robustness of the field network we start the recall in a scene that lacks the second object of the sequence. In t0 and t1, the network's activation develops analog to the previous two recall examples with a color slice used to extract the target position and the position match to determine the successful termination of the movement. However as the second ordinal node activates at t2 no blue object is present in the scene, thus no sub-threshold activation blob overlaps with the blue color slice in the Memory Color Selection field and no peak forms.At points t3, the blue object is added to the scene, which is committed to memory and afterwards extracted as a valid target position. The movement than concludes at t4 with the arm occluding the purple object, which is kept in working memory due to the self-sustaining kernel. The working memory information is then used in t5, when the third ordinal node specifies purple as the next sequence color. Thus the sequence ends at t6 with no further perturbations.At point We have presented a network of dynamic neural fields that integrates the complete pathway from the sensor surface (vision) to representations of higher cognition  and to the motor system (pointing). The network architecture enables a robotic agent to autonomously learn a sequence of colors from demonstration and then to act according to the defined serial order on a scene. Both during learning and while acting out the sequence, the transitions between elements of the sequence are detected without the need for an external control signal .In each of the three sub-networks responsible for scene representation, the representation of serial order, and movement generation, sequential transitions between neural activation states are brought about through the mechanism of the condition of satisfaction. Thus, visual attention shifts only once a currently attended item has been committed to working memory. A transition to the next element in the serial order occurs only once the robot has successfully acted on the current element. And an arm movement terminates only once the desired movement target has been reached. The mechanism of the condition of satisfaction thus reconciles the capacity to autonomously act according to learned or structurally determined plans with the capacity to be responsive to sensory or internal information about the achievement of goals.The scenario was simple, but meant to demonstrate the fundamental components of any neurally grounded autonomous robot.A representation of the visual surround is the basis for any intelligent action directed at the world. It is also the basis for sharing an environment with a human user. We humans are particularly tuned to building scene representations which form the basis of much of our visual cognition  and from time (when is each item addressed). Based on these abstractions, a broad set of actions can be conceived of as serially ordered processing steps. For instance, assembling a piece of IKEA furniture could be described this way. Unlike many classical, disembodied cognitive tasks, real action sequences require the capacity to deal with variable and perhaps unpredictable amounts of time needed to achieve each processing step. Learning the—a priori—arbitrary contents of a serially ordered sequence makes this scenario quite powerful. It goes beyond, for instance, a mere capacity to imitate or emulate behavior, which would lead to the reproduction of the same movements or effects without generalization to new conditions. It also goes beyond the generation of sequences of behaviors that would be triggered by environmental conditions according to a fixed organizational scheme encoded in a behavior-based robotic architecture .The scene representation system was built on a single, trivial color feature. We have previously explored and demonstrated elsewhere how a neural dynamic system of the same kind can deal with multiple feature dimensions , and if action was richer , then the modeled interface would already make the robot quite useful. It would enable a robot to learn the solution of problems from a human user, as long as the perception system extracts the conceptual structure of the demonstrated action. A big extension would be the capacity of the system to solve problems by itself, devising the sequences of actions required to achieve a goal. This would require neural processes in new domains such as exploration, outcome representations, perhaps value systems. There is a growing literature on such models  is contained within the manuscript. For technical questions regarding the software please contact the authors.All relevant data to reproduce the presented dynamic field network with our open-source software cedar (JT and AF performed the work, and contributed to the writing of the manuscript. YS and GS conceived of the project, provided research supervision, and contributed to the writing of the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The automatic assessment of music performance has become an area of increasing interest due to the growing number of technology-enhanced music learning systems. In most of these systems, the assessment of musical performance is based on pitch and onset accuracy, but very few pay attention to other important aspects of performance, such as sound quality or timbre. This is particularly true in violin education, where the quality of timbre plays a significant role in the assessment of musical performances. However, obtaining quantifiable criteria for the assessment of timbre quality is challenging, as it relies on consensus among the subjective interpretations of experts. We present an approach to assess the quality of timbre in violin performances using machine learning techniques. We collected audio recordings of several tone qualities and performed perceptual tests to find correlations among different timbre dimensions. We processed the audio recordings to extract acoustic features for training tone-quality models. Correlations among the extracted features were analyzed and feature information for discriminating different timbre qualities were investigated. A real-time feedback system designed for pedagogical use was implemented in which users can train their own timbre models to assess and receive feedback on their performances. In recent years, several computational systems have been developed with the aim of enhancing music education and instrument tuition. In these systems automatic assessment of musical performance plays a central role. However, human assessment is often subjective, thus making the implementation of an automatic assessment system a significant challenge. In music education, assessment relies on consensus of highly trained experts who produce subjective interpretations of performance  quality and its implications on the quality of music performance. Saitis et al.  studied Several studies have investigated the verbal description and/or components of tone and timbre quality. Research aiming to obtain representative timbre spaces have been conducted in the past by means of perceptual similarity experiments Grey, . StudiesFrom the computational perspective timbre has been studied in terms of its relation to the audio descriptors that can be mathematically computed from the digital audio signal. In general, machine learning techniques are used to find patterns that permit the recognition of different timbre qualities from the descriptors extracted from the audio signal  across the violin tesitura (see section 3.1.2), always using the same four notes for each of the defined tonal quality dimensions.Participants: There were 20 respondents  with a mean age of 36.49 years . They had a mean musical experience of 11.8 years , with representation from professional (23%), student (19%), and amateur (55%) groups and 100% having taken formal lessons on their primary instrument for a mean 5.15 years . The cohort was represented 8 nationalities, with a significant proportion being Spanish (63%). The range of primary instruments included violin (36%) and guitar (35%) with the remaining (29%) comprising a mix of percussion, vocal, and other instruments. One third (34%) of the cohort reported classical as their primary genre, with the remaining comprising jazz, folk, pop, and other. The survey opened with an information sheet outlining the topic and purpose of the study and instructing respondents that, by beginning the survey, they were providing informed consent. Ethical approval for the study, including consenting procedures, was granted by the Conservatoires UK Research Ethics Committee following the guidelines of the British Psychological Society.Procedure: Respondents were prompted with an initial page with written instructions followed by a form to collect demographic information and musical background information. A form was then presented containing a sound player with the corresponding sound sample and a list of the tone quality dimensions . Data filtering was performed using an interquartile range (IQR) filter with an extreme value factor set to three times the IQR and an outlier value factor of 1.5 times the IQR.Audio descriptors help to characterize an excerpt of an audio signal (or an audio frame) in terms of its spectral, temporal, and spectro-temporal properties. Audio descriptors are divided into global and instantaneous descriptors. The former refer to those in which the whole signal is used for its computation , the latter are computed at a frame-level for each frame at a time. For each of the audio excerpts, we extracted frame-based low-level audio descriptors . On one hand, real-time feedback requires fast descriptor computation and thus frame based descriptors are a natural choice. On the other, spectral descriptors are obtained from the information contained in the spectrogram of the audio wave, which has a direct relation with timbre. Global descriptors were considered as well by computing the mean and the standard deviation of the frame-based descriptors over a sliding window of 600 ms. The computation of these aim to encode information on the stability of the studied timbre qualities of a performed note over time.Automatic feature selection using filter methods was used to obtain the subset of features most relevant for classification. Filter methods use a proxy measure  to score features, where these are filtered and ranked by information gain values. We used the methods provided by the Weka library  with radial kernel, and Artificial Neural Networks (ANN) with one hidden layer . Offline tests were performed using the Weka machine learning library .Classification experiments were carried out over the expert-defined tone qualities subset. Our aims were to obtain classification models for the defined paired labels for tone quality and to obtain a subset of the audio features that best predicted each of the aforementioned tonal qualities. We considered two main approaches for the modeling process:By pitch range: The aforementioned modeling strategies were carried out on instances grouped by pitch range. We considered a distribution of low, medium, and high registers. This subdivision was done by octaves over the violin register, i.e., the first octave (from G2 to G3) was considered “low,” second octave (from G3 to G4) was considered “medium,” and notes above G4 were considered “high” register.By position: The modeling strategies took into consideration whether notes were played in the 1st or 5th position of the violin. This distribution resulted from how the recordings were played by the violinist. By finger: The modeling strategies were used in subsets defined by the finger used to play a particular note.Several sub-groupings were considered to test the consistency of the obtained models across several scenarios as follows:The subjective nature of timbre/tone perception and label semantics produces several complications for obtaining a predictive model to generalize to different performers and different performance conditions . As a result, the motivation of a user-defined machine learning approach was to offer a tool able to classify audio samples in real-time based on training examples given by a user, where semantic labels of tone quality can be user-customized.Based on the results of feature selection from the offline approach (see section 3.2.2), for this approach we considered the features presented in For the user-defined modeling approach we used the same three machine learning methods mentioned in section 3.2, where the ANN was set as default based on the offline machine learning analysis (see section 4.3). The system permits the storage of the recorded data of the computed features along with their respective user-defined labels as well as the generated models. A detailed explanation on the real-time implementation is presented in the results section 4.4.Consistency among participants' ratings was assessed using Cronbach's coefficient  and is presented in Previous results showed high correlations among similar opposite scales: e.g., narrow/broad and restricted/free see . SimilarListeners reported having perceived that some of the adjectives  correlated with the quality of violin/microphone used for recordings. A future approach would obtain recordings on the same set of tone qualities by different performers/instruments. Listeners also indicated that other adjectives such as restricted/free and narrow/broad were perceived with relation to the dynamics of the performance. Similarly, some mentioned that dimensions such as broad/narrow and open/close were related to the level of vibrato. Closer study of these aspects will be regarded in future work.The improvement in accuracy in training the tone quality models over different feature subsets, ranging from 1 feature to all features  was addressed see . The whot-test (p < 0.05) showed significant improvement over the baseline in all of the scenarios considered. Similarly, no relevant differences in the accuracy of prediction of the models were observed across the studied scenarios. In The results of the different scenarios and sub-groupings taken into consideration are summarized in The implementation of the real-time tone quality feedback system is embedded in SkyNote, a general system for automatic assessment of violin performance developed within the scope of the TELMI project  semantic labels, e.g., rich and poor timbre labels. We then compared the accuracy of the user-generated timbre models with that of the “pre-trained” models.Four professional violinists were asked to record audio examples of rich/poor timbre notes at the four different registers explained in section 3.2, as well as one paired example of a self-defined timbre quality dimension. Each example consisted of a pair of notes, one with the opposite semantic label of the other one which was used as training data. After training users tested the real-time feedback system and recorded again one opposite-label pair of notes for testing. We recorded both the data obtained in real time and the models trained by each subject. We applied Artificial Neural Networks (which produced the best accuracy as seen in section 4.3) for evaluating the obtained models. The average accuracy obtained on the models is presented in We have presented a machine learning approach for the automatic assessment of the quality of tone in violin performance. We have obtained a list of 10 opposite semantic tonal dimensions provided by violin experts. We obtained recordings of each of the tonal dimensions performed by a professional violinist and performed a listening test of the provided dimensions using an online survey. The semantic associations of listeners over the defined tonal dimensions were studied based on inter-user and inter-dimension correlations. Spectral low-level descriptors were extracted from the recording examples to later train machine learning models. An offline machine learning approach was performed to investigate the accuracy obtained with three different learning schemes, as well as across several performance scenarios . A subset of features was selected for a real-time approach, where the system extracted in real-time the aforementioned set of descriptors and provided real-time feedback on the quality of the proposed tonal dimensions. The system is able to be re-trained with user-defined sound examples and semantic labels. An evaluation of the accuracy of the user-trained models was performed in which it was observed that the selected set of features contained enough information to correctly classify different intended performed tonal qualities.In general, participants in the perceptual study pointed out that the differences across the different tone examples were so subtle that it was difficult to remember previous samples, which may explain the lack of consensus among participants. However, computers have perfect memory and are able to extract features characterizing the acoustic features of the audio samples which seem to be informative for differentiating the samples. Furthermore, in the cross validation tests among performers, some participants managed to tune the performance resources in order to match the tonal quality model trained by a second violinist. The majority of existing studies have looked at very different tones either synthetically generated or across instruments. However, given the fact that in this study we are considering several variables simultaneously  it might be the case that, in the real world, the differences among intended performance tonal qualities are often quite subtle.Most of the systems reviewed in section 2 approach the correctness of a performance by assessing its timing and pitch accuracy, whereas the system presented in this paper deals with the quality of sound produced by the performer. In this sense, the work by Romani Picas et al.  is the mInitial versions of our system were also trained using recordings made under specific conditions  and, as the system proposed by Romani Picas et al. , failed The similarity of some of the considered tonal dimensions might be better addressed on a comparative type test, where users can provide a rating of a particular tone quality based on the possibility of listening to several audio samples. Similarly, a large scale study with violin performers could be performed to obtain the semantic labels following a similar methodology used by Saitis et al. . For theThe evaluation of the system and its user-defined approach, including implications for practice in music education contexts, will be addressed in future work.SG, RR, GW, AP, and AW contributed conception and design of the study. SG, AP, RR, GW, and AO were involved the data acquisition. SG, GW, and AO designed and programed the platform for the survey data acquisition. SG performed the statistical analysis. OM led the main software implementation framework. IN and SG performed the main programming tasks. SG wrote the first draft of the manuscript. All authors contributed to manuscript revision.The handling editor is currently editing co-organizing a Research Topic with one of the authors AW, and confirms the absence of any other collaboration. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.