Pumps are one of the core components of drilling equipment, and their fault diagnosis is of great significance. The data-driven approach has made remarkable achievements in the field of pump fault diagnosis; however, most of them are easily affected by complex background conditions and usually suffer from data scarcity problems in real-industrial scenarios, which limit their application in practical engineering. To overcome the above shortcoming, a novel framework for a model named Hyperparameter Optimization Multiple-Signal Fusion Transfer Convolution Neural Network is proposed in this paper. A convolutional neural network model based on transfer learning is built to promote well-learned knowledge transfer over different background conditions, improve robustness, and generalize the model to cross-domain diagnosis tasks. The multi-signal fusion strategy is involved in capturing system state information for establishing the mapping relationship between the raw signal and fault pattern by integrating the multi-physical signal with the weight allocation protocol. The hyperparameter optimization method is explored in conjunction with the transfer-based model by integrating Grid Search with the Gradient Descent algorithm for further improvement of diagnosis performance. Results show that the proposed model can effectively realize the fault diagnosis of pumps under different background conditions, achieving 95% accuracy. With the increasing demand for natural resources in human society, drilling rigs play a significant role in the extraction process of petroleum and gas resources. The pump is one of the core components of the drilling rig, which is responsible for converting mechanical energy into hydraulic energy and providing hydraulic oil to the drilling rig . During At present, research on fault diagnosis has focused on the data-driven approach , which dThe signal types required for fault diagnosis of the pump based on the signal processing method include vibration signals  and sounThe fault diagnosis method of pumps based on artificial intelligence has achieved long-term development due to the wide application of deep learning. The fully connected neural network models, such as the deep belief network , the deeFor example, Yan et al.  proposedAlthough the CNN methods significantly improve performance for pump fault diagnosis, those normal approaches still pose two potential problems that limit their application in practical engineering. More specifically, the background conditions of the pump will inevitably change, which means it is difficult for the established neural network model to generalize the fault pattern knowledge from the labeled training data to the unlabeled testing data due to the interference of variable background conditions. In addition, real industrial scenarios pose challenges to the collection of data, which often leads to the problem of data scarcity. Affected by safety and economic factors, data collection is generally carried out only for a specific period of time, which limits the amount of data. The time distribution of construction machinery in different working states is not average; therefore, the collected data are unbalanced. A complex background environment will also have a negative impact on the quality of the data. Since the quality, distribution, and even quantity of sampled data from different background conditions cannot be guaranteed, this inevitably increases the difficulty of neural network models in cross-domain diagnostic tasks and reduces the usability of the method in real engineering scenarios.In this condition, the transfer learning strategy, which has the capability to reuse the well-learned knowledge from one condition to another by fine-tuning operations, is beneficial for alleviating the above problem. From the perspective of background conditions, due to the advanced learning of common knowledge in different backgrounds, the impact of background changes is only caused by characteristic knowledge, and the sensitivity of neural network models to background conditions is significantly reduced. From the perspective of data scarcity, due to the repeated use of knowledge across conditions, high-quality data under certain conditions can provide support for learning tasks under all conditions, thereby reducing the dependence of neural network models on large amounts of training data. Therefore, the transfer learning strategy can reduce the learning cost by reusing the knowledge obtained from the source task in the training of the target task. The reduction of the learning cost makes the transfer learning strategy available on an edge device with limited resources, and many researchers have involved this technology in fault diagnosis fields.Nevertheless, to the best of our knowledge, no research has been introduced for pump fault diagnosis in a real-industrial scenario so far. However, similar research based on transfer learning for rotating machine fault diagnosis tasks has been widespread in recent years, which may provide an effective reference for fault pattern detection of pumps in real-industrial environments. For example, Tang et al.  proposedAlthough the current transfer learning models can improve diagnosis performance to some extent, most of them only rely on a single physical signal. Compared with the multiple types of physical signals, a single physical signal may not be able to fully describe the complex working state of a machine such as a pump. In this way, the extracted feature from the collected single physical signal cannot form sufficient information representation to make it susceptible to random factors, which limits the robustness and generalization of the transfer learning model for fault diagnosis under variable background conditions.In addition, the hyperparameter setting of the transfer learning model is an important problem, and the proper combination of hyperparameters can improve the performance of the transfer learning model. The manual setting of hyperparameters will be affected by the operator experience, and the number of attempts is relatively limited; therefore, it is not easy to obtain the optimal combination of hyperparameters. Compared with manually setting the hyperparameters of the transfer learning model, automatic tuning of the hyperparameter combination presents great advantages . The optTo solve these problems, a novel framework for a model named Hyperparameter Optimization Multi-Signal Fusion Transfer Convolution Neural Network (OMTCNN) is proposed in this paper. A new transfer-based convolutional neural network model is first designed to promote well-learned knowledge transfer over different background conditions for improving robustness and generalizing the model in cross-domain diagnosis tasks. Then, the multi-signal fusion strategy is involved to capture the state information of the pump for establishing the mapping relationship between the raw signal and fault pattern by integrating the multi-physical signal with the weight allocation protocol. Afterward, a hyperparameter optimization method is explored to arm with the transfer learning model by integrating Grid Search with the Gradient Descent algorithm for further improving diagnosis performance. By doing this, the OMTCNN model can effectively realize the fault diagnosis of pumps under different background conditions.Strengthen the ability of convolutional neural networks to diagnose pump faults under different background conditions by introducing transfer learning.Break through the limitation of a single type of signal for pump fault diagnosis by using multiple types of signals for multi-signal fusion.Optimize the fault diagnosis results of the pump through the automatic setting of hyperparameters.The main contributions of the network model proposed in this paper are as follows:The remainder of this article is organized as: A convolutional neural network  is a feeThe convolutional layer consists of neurons with learnable weights and bias constants, which are used to convolute the input data to realize feature extraction. The convolution operation can be expressed as follows:The convolution operation mainly involves two important concepts, namely, sparse connection and weight sharing. Sparse connection refers to the fact that the input data and the convolution layer are not fully connected but create connections in the local area of the convolution layer, realizing the extraction of the input data features through the local perception of the convolution kernel. The weight sharing obeys the assumption that the importance characterization results of input data features from the same dimensional plane should be consistent. In other words, the weights and bias constants of the same convolution kernel remain constant. Weight sharing can effectively reduce the number of network training parameters by means of convolution kernel check feature repeated recognition, without considering the feature location distribution.The pooling layer aggregates the input data region into the output of the feature map through the region adjacency of the feature mapping, which is used to describe the association between the output and the region, and realizes the down-sampling processing of the input data. Down-sampling processing can improve network efficiency and avoid the overfitting phenomenon. The pooling layer is also used to reduce the sensitivity of the convolutional layer to the target position so that its feature extraction is not seriously affected by the change in the target position. The pooling layer generally adopts the maximum pooling strategy or the average pooling strategy, which can be stated by the following formula:The fully connected layer maps the features extracted from the convolution layer and the pooling layer to the sample space. These features can be obtained by the convolutional layer and the pooling layer mapping the input data to the feature space. Ignoring the influence of spatial structural features after feature flattening, combining all local features of input data into global features, and achieving the representation of input data features by output.The activation function is a nonlinear function that acts on the output of each neuron in the convolutional layer and the fully connected layer. The activation function introduces non-linear factors to the network so that it can approach almost arbitrary functions and improve its expression ability.Dropout is a regularization method that randomly and temporarily discards some neuronal nodes with a certain probability in network training. It is mainly used to solve the overfitting problem that the accuracy of the network is high in the training set and low in the testing set, so as to improve the stability of the network.Convolutional neural networks are trained through forward propagation of input data, back propagation of the loss function representing output errors, and updating of network parameters. The forward propagation calculation can be expressed by the following formula:Transfer learning  is a metDomain, which can be denoted as Task can be denoted as Transfer Learning, given a source domain The high-dimensional connection obtained by the convolutional neural network model in the learning task in a specific background condition, that is, the mapping of the source domain features to the sample space, has a guiding effect on the relevant learning task in other background conditions. The knowledge obtained in the source domain learning task can be used to help the learning of the corresponding task in the target domain by using transfer learning. This reduces the interference of background condition changes in the convolutional neural network model. From the perspective of data volume, since the convolutional neural network model obtains the common knowledge of the relevant learning tasks of the source domain in advance of the learning tasks of the target domain, the learning of the relevant tasks of the target domain mainly focuses on the unique knowledge related to the background condition, and the demand for the amount of data will be relatively reduced. Therefore, a convolutional neural network model based on transfer learning overcomes the shortcomings of the traditional convolutional neural network model.In transfer learning, different techniques are applied to convolutional neural network models, and typically they are a specific combination of pre-training, freezing, fine-tuning, and adding new layers. A network model trained with source domain data are called a pretrained network model, consisting of pretrained layers. Freeze and fine-tuning are techniques that use some or all layers of pre-trained network models to train on a target domain. Freezing certain layers means that the trainable parameters do not change and are constant for frozen layers in pre-trained network models. Fine-tuning means initializing trainable parameters with pre-trained layers rather than randomly initializing the entire network model or some selected layers. Another new technique is based on freezing a pre-trained network model and adding a new layer to that model to train on target data.The strategy of freezing the first several layers of the convolutional neural network model can be used to carry out transfer learning, which can further demonstrate the advantages of the convolutional neural network model based on transfer learning. According to the characteristics that convolutional neural networks extract low-dimensional features with shallow layers from data and convolutional neural networks extract high-dimensional features with deep layers from data, the strategy of freezing the shallow layers of convolutional neural networks can be used to carry out transfer learning. The common knowledge of the learning task under different background conditions is learned by the shallow layer of the convolutional neural network, and the common knowledge is fixed by freezing the shallow layer of the convolutional neural network. The characteristic knowledge of learning tasks under different background conditions is learned through the deep layer of the convolutional neural network, and the frozen shallow convolutional neural network is carried out through transfer learning to complete the effective extraction of high-dimensional features of data. Furthermore, because the convolutional neural network after transfer learning mainly extracts high-dimensional features of data, the demand for data volume will be relatively reduced. Therefore, the pump fault diagnosis results of the convolutional neural network model based on transfer learning under variable background conditions have better performance than the traditional convolutional neural network model.In the process of data collection, multiple types of signals can be collected by installing various types of sensors. Compared with a single type of signal, multiple types of signals can describe the pump system from different angles, which can better reflect the essential characteristics of the pump system, increase resistance to random factors, and improve robustness. However, traditional convolutional neural network models are not good at directly processing data of multiple signal types, and the effect of pump fault diagnosis is limited. In order to solve this problem, a multi-signal fusion module based on weight allocation is proposed in this paper. The structural diagram of the multi-signal fusion module is shown in The multi-signal fusion module based on weight allocation mainly includes three parts: standard normalization, reliability operation, and weight setting. The module is used to fuse a group of signal samples of multiple types into a fusion sample containing multi-signal-type information. The standard normalization part is responsible for adjusting the probability distribution of multiple types of signal samples to the same statistical space for subsequent calculations. In the reliability operation part, the reliability degree of each signal sample of different types is obtained by means of convolution operation, global average pooling operation, and classification operation. The weight-setting section makes weight adjustments according to the reliability of each signal sample of different types. Finally, the multi-signal fusion module combines a group of signal samples of multiple types after weight setting into a fusion sample containing multi-signal type information. The fusion sample of multiple signal types can characterize the pump system more appropriately and accurately than the original signal of multiple types. Therefore, the multi-signal fusion module based on weight allocation can realize the pump fault diagnosis of multiple signal types.Different types of sensors can collect different types of signals to reflect the operating condition of the pump, such as vibration signals , pressurThe A/D converter converts multiple types of signals into a data matrix that characterizes the pump’s operating state from different angles, including the vibration data matrix, pressure data matrix, and flow data matrix. The multi-signal fusion module based on weight allocation is used to process the data matrix.Firstly, the numerical distribution of the vibration data matrix, pressure data matrix, and flow data matrix is adjusted by standard normalization as follows:Secondly, the reliability of the vibration data matrix, pressure data matrix, and flow data matrix are verified, respectively. After the data matrix passes through the convolution layer, the global average pooling layer, and the Softmax classifier, The calculation process for Softmax is as follows:Finally, according to the relative value of the reliability probability of the vibration data matrix, pressure data matrix, and flow data matrix, the respective weight ratio is obtained as follows:The vibration data matrix, pressure data matrix, and flow data matrix are weighted and fused into a fusion matrix. And match the corresponding Loss function:The multi-signal fusion module based on weight allocation improves the input quality and reverse update of the convolutional neural network model based on transfer learning, which has a positive effect on the fault diagnosis of pumps.When the traditional convolutional neural network model solves the problem of pump fault diagnosis under different background conditions, the fault diagnosis accuracy of the traditional convolutional neural network model is relatively limited due to the influence of background condition changes and the fact that the amount of data in all background conditions may not be sufficient. To solve the problem, it is proposed to develop a convolutional neural network model based on transfer learning, which takes advantage of transfer learning to solve the pump fault diagnosis of a convolutional neural network model under variable background conditions. The structural diagram of the convolutional neural network model based on transfer learning is shown in The basic convolutional neural network model consists of three convolutional layers, a maximum pooling layer, and three fully connected layers, and sets the ReLU activation function and Dropout.The application of transfer learning in convolutional neural networks involves a variety of frozen strategies. For example, freezing all layers except the last layer of the pre-trained network for the dataset in the source domain, no longer updating the training network parameters of the frozen layer, and training the final layer of the pretrained network using the dataset in the target domain may also freeze some layers of the pre-trained network for the dataset in the source domain or not to perform freezing operations, and use the dataset in the target domain to train some or all layers of the pre-trained network.In this paper, the convolutional neural network model based on transfer learning is used to diagnose pump faults under different background conditions. Firstly, a relatively suitable background condition is selected as the source domain, and the data under this condition is used to train the convolutional neural network model. Secondly, the first two layers of the trained convolutional neural network model are frozen. Finally, the first two layers of a frozen convolutional neural network model are used to carry out fault diagnosis under other background conditions, that is, the target domain, and realize transfer learning under different background conditions. The convolutional neural network model based on transfer learning can realize transfer learning under any background condition, so as to realize pump fault diagnosis under variable background conditions.The setting of hyperparameters will affect the performance of the convolutional neural network model, and a reasonable setting of hyperparameters can effectively improve the accuracy of fault diagnosis in the convolutional neural network model. The method of manually setting hyperparameters is easily dictated by empirical factors; therefore, the difference between the experimenters who set hyperparameters may affect the result of the fault diagnosis. Moreover, due to the limited efficiency of manually setting the hyperparameters, it is often impossible to obtain a relatively appropriate combination of hyperparameters. To solve this problem, a hyperparameter optimization module based on the algorithm integrating Grid Search with Gradient Descent is proposed in this paper.The relatively important hyperparameters of the convolutional neural network model mainly include batch size and learning rate. The hyperparameter optimization module can be used to find the optimal combination of batch size and learning rate within a certain range. For the batch size, its distribution is discrete, and the optimal selection is usually concentrated on partial values; therefore, the grid search method can be used to find the most optimal batch size. As for the learning rate, its distribution is continuous and the minimum value is few; therefore, the method of gradient descent can be used to find the optimal choice of learning rate. The hyperparameter optimization module reduces the impact of the differences between the experimenters on the results of fault diagnosis and can find a set of relatively suitable hyperparameter combinations in the two-dimensional space composed of batch size and learning rate. Therefore, the hyperparameter optimization module based on the algorithm integrating Grid Search with Gradient Descent can optimize pump fault diagnosis results.The following methods can be used to select relatively suitable hyperparameter combinations: According to the optimal distribution characteristics of batch size, an optimization range is determined for batch size, and an appropriate value is found in the optimization range to assign a hyperparameter. The optimization range of batch size is composed of discrete values that fit the grid search. It can be represented as follows:In the two-dimensional space composed of batch size optimization ranges and learning rate optimization ranges, the algorithm integrates Grid Search with Gradient Descent. That is, the value in the optimization range of a batch size is fixed, the gradient descent is carried out in the optimization range of the learning rate, and then the operation is traversed through all the values in the optimization range of the batch size. In order to increase the probability of finding the optimal value, multiple gradient descents can be performed by random means of multiple initial values. In addition, a break-out mechanism is designed for the hyperparameter optimization module. When the output accuracy of the convolutional neural network model does not reach a new high after several iterations, the iteration is interrupted, and then other numerical combinations are verified. By comparing the combinations of values in the optimization space, a set of relatively suitable values is obtained to assign hyperparameters. The block diagram of the hyperparameter optimization module is shown in The main optimization process of the hyperparameter optimization module is as follows:Step 1. Running the Grid Search algorithm within the preset range of batch size yields several batch size values.Step 2. For each batch size, run the Gradient Descent algorithm multiple times within the preset range of the learning rate to obtain several learning rate values.Step 3. Several combinations composed of specific batch sizes and corresponding learning rates to obtain the local optimal combination.Step 4. Screen out the global optimal combination from all the local optimal combinations.The hyperparameter optimization module based on the algorithm integrating Grid Search with Gradient Descent can automatically set relatively appropriate hyperparameters for the convolutional neural network model based on transfer learning, which provides important help for the fault diagnosis of the pump.The model of OMTCNN is presented in The convolutional neural network model based on transfer learning is mainly employed for realizing the transfer training of the convolutional neural network model under different background conditions. The multi-signal fusion module is mainly applied for the data matrix fusion of vibration data matrix, pressure data matrix, and flow data matrix. The hyperparameter optimization module is mainly used for the automatic setting of batch size hyperparameters and learning rate hyperparameters. The operational process of the OMTCNN model is shown in The main operational process of the OMTCNN model in fault diagnosis is as follows:Step 1. Obtain the fusion data matrix from the original data through the multi-signal fusion module.Step 2. Through the hyperparameter optimization module, automatically set hyperparameters for the convolutional neural network model based on transfer learning.Step 3. The convolutional neural network model based on transfer learning obtains fault diagnosis results in the source domain.Step 4. Put the fusion data matrix of the target domain into a convolutional neural network model based on transfer learning.Step 5. The convolutional neural network model based on transfer learning performs transfer learning to obtain fault diagnosis results in the target domain.The pump consists of five cylinders, each containing two valve bodies, namely the suction valve and the release valve. This paper makes four classifications on the fault status of the cylinder according to the failure of the suction valve and the release valve, and the four types of fault status are normal double valve, suction valve failure, release valve failure, and whole valve failure. These four types of fault status correspond here to N, S, R, and W.During the operation of the pump, the background conditions, such as the stroke of the pump and the pressure of the pump, will change. The stroke of the pump is mainly maintained at 90 strokes per minute (SPM) and 110 SPM, and the pressure of the pump is mainly maintained at 20 Mpa and 40 Mpa. The conditions of the stroke of the pump and the pressure of the pump can be divided into four situations: 90-20, 90-40, 110-20, and 110-40. In this paper, these four situations correspond to A, B, C, and D conditions. Vibration sensors, pressure sensors, and flow sensors are used to collect the relevant signals from the 1# cylinder and the 2# cylinder of the pump under A, B, C, and D conditions. The signal collection of the five-cylinder pump on site is shown in The vibration signal, pressure signal, and flow signal are converted by an A/D converter to obtain vibration, pressure, and flow data. Specifically, the sampling frequency of the three types of sensors is 1000 Hz, and the three types of data generated every 1 s is intercepted as a sample. 100 samples with vibration, pressure, and flow data can be obtained each under A, B, C, and D conditions in a 1# cylinder and a 2# cylinder. The vibration data samples, pressure data samples, and flow data samples under specific conditions are arranged according to the collection sequence. Then, a single sample of vibration data, pressure data, and flow data with the same sequence is selected for fusion operation in sequence, and several fusion samples corresponding to the sequence are obtained. Each fusion sample contains vibration, pressure, and flow information. All fusion samples under a specific condition are randomly allocated to form a training set with a proportion of 0.7 and a test set with a proportion of 0.3 and are extended to all conditions to complete the data preparation.The following experiments were conducted under the Linux-based Ubuntu operating system using the Python programming language in the PyTorch integrated development environment.In order to evaluate the benchmark performance of the basic CNN model, as shown in The accuracy of using the basic CNN model for a 1# cylinder and a 2# cylinder under A, B, C, and D conditions is shown in From From In order to demonstrate the improvement of performance through the introduction of transfer learning, this experiment uses the transfer convolutional neural network (TCNN) model to diagnose faults within the 1# cylinder and 2# cylinder, respectively, under the conditions of A, B, C, and D, with any background condition as the source domain and other background conditions as the target domain. The TCNN model employs the transfer learning function of the basis CNN model, and its parameter configuration is consistent with the basis CNN model. The accuracy of using the TCNN model for 1# cylinder and 2# cylinder under the conditions of A, B, C, and D is shown in It can be observed that the TCNN model has a significant improvement in fault diagnosis accuracy compared to the basic CNN model under A, B, C, and D conditions for internal transfer of the 1# cylinder and 2# cylinder, and the fluctuations under A, B, C, and D conditions have been weakened. In further experimentation, the TCNN model performed fault diagnosis on the A, B, C, and D conditions between the 1# cylinder and the 2# cylinder. Through transfer learning, using any background condition as the source domain and other background conditions as the target domain, the experimental results are shown in It can be observed that the fault diagnosis accuracy of the TCNN model in the transfer between the 1# cylinder and the 2# cylinder under A, B, C, and D conditions is generally lower than the accuracy of the internal transfer of the cylinder; however, it still has a certain improvement compared to the basic CNN model and controls the waves under A, B, C, and D conditions. The experiment result indicates that, compared with the basic CNN model, the TCNN model improves generalization ability through transfer learning and maintains a certain degree of robustness. Next, attempt to improve the TCNN model’s processing capacity for multiple types of signals by introducing a multi-signal fusion module.In order to test the performance improvement brought by processing multiple types of signals in the multi-signal fusion module, this experiment uses the multi-signal fusion transfer convolution neural network (MTCNN) model to perform fault diagnosis using transfer learning under the conditions of A, B, C, and D within the 1# cylinder, with any background condition as the source domain and other background conditions as the target domain. The MTCNN model has added a multi-signal fusion module on top of the TCNN model, and its parameter configuration is inherited. Select transfer network models with a structure scale similar to the proposed method are shown in In addition, based on the same or similar model structure and parameter configuration as the related transfer network model, using the same dataset for the experiment is shown in It can be observed that the MTCNN model has certain advantages in fault diagnosis accuracy compared to other transfer network models in the internal transfer of the 1# cylinder under A, B, C, and D conditions. Further experimentation uses the MTCNN model through transfer learning to diagnose faults in the A, B, C, and D conditions of the 1# cylinder, also using any background condition as the source domain in the 2# cylinder and other background conditions as the target domain in the 1# cylinder. And compare it with other transfer network models shown in It can be observed that the MTCNN model still has advantages in fault diagnosis accuracy compared to other methods under A, B, C, and D conditions when transferring from a 2# cylinder to a 1# cylinder. However, computation efficiency is also an important indicator for measuring the performance of network models. The computation efficiency of the MTCNN model and other transfer network models on the specific dataset is shown in It can be observed that, compared with other methods, the MTCNN model ensures computation efficiency in the process of fault diagnosis. And if only considering the fault diagnosis results of the internal transfer of 1# cylinder and the transfer from 2# cylinder to 1# cylinder under A, B, C, and D conditions, the MTCNN model achieves optimization based on the TCNN model. In addition, the MTCNN model was used to further test the fault diagnosis results using different types of signals, as shown in It can be observed that the MTCNN model has the best performance when using three types of signals. The experiment result indicates that, compared with the TCNN model, the MTCNN model uses the feature extraction ability to extract features with relatively high reliability by setting the weight of multiple types of signals and enhancing processing capability for multiple types of signals. On this basis, try to continue optimizing the fault diagnosis results with the automatic setting of hyperparameters by introducing the hyperparameter optimization module.In order to verify the optimization results of fault diagnosis through the introduction of the hyperparameter optimization module, this experiment uses the OMTCNN model to diagnose faults through transfer learning within the 1# cylinder under A, B, C, and D conditions, using any background condition as the source domain and other background conditions as the target domain. The OMTCNN model has added a hyperparameter optimization module on top of the MTCNN model, and its parameter configuration is consistent with the MTCNN model. The accuracy of OMTCNN under A, B, C, and D conditions of internal transfer in a 1# cylinder is shown in Further experimentation using the MTCNN model to diagnose faults in the A, B, C, and D conditions of the 1# cylinder through transfer learning using any background condition in the 2# cylinder as the source domain and other background conditions in the 1# cylinder as the target domain The accuracy of the OMTCNN model under A, B, C, and D conditions from 2# transfer to 1# is shown in It can be observed that, compared to the MTCNN model, the OMTCNN model has higher fault diagnosis accuracy in the internal transfer of the 1# cylinder and the transfer from the 2# cylinder to the 1# cylinder under A, B, C, and D conditions.In order to explore the performance improvement of the OMTCNN model compared to the MTCNN model, statistical analysis of relevant indicators of OMTCNN and MTCNN is required, as shown in It can be observed that OMTCNN leads MTCNN in relevant statistical indicators. In order to further explore the potential performance improvement of the OMTCNN model compared to the MTCNN model, as shown in According to the confusion matrix analysis, firstly, the diagnostic accuracy of the OMTCNN model is higher than that of the MTCNN model for all fault status types. Secondly, the OMTCNN model has more fault status types with no diagnostic errors than the MTCNN model. Finally, the fault status types that the OMTCNN model experiences misjudgments are strictly within the fault status types that the MTCNN model experiences misjudgments. Reflects the advantages of OMTCNN in terms of diagnostic accuracy.According to the analysis of the training process, firstly, at the same number of epochs, compared to the MTCNN model, the accuracy of the OMTCNN model is often higher and the Loss is often lower. Secondly, under the same accuracy or Loss, the OMTCNN model often has fewer epochs compared to the MTCNN model. Finally, the OMTCNN model did not exhibit any significant accuracy fallback phenomenon as observed in the MTCNN model. Reflects the advantages of OMTCNN in terms of computational efficiency.The experiment result indicates that, compared with the MTCNN model, the OMTCNN model improves the ability to extract features by automatically setting the hyperparameters and achieves performance improvement. In addition, the t-SNE of inputs Cov1, Cov2, and Cov3 for describing the performance of the OMTCNN model is provided as shown in It can be observed that through the OMTCNN model, the characteristics of different fault status types can be clearly separated. Therefore, the OMTCNN model can achieve fault diagnosis for pumps.In this article, a novel OMTCNN model is proposed for pump fault diagnosis under different background conditions. First, a new transfer-based convolution neural network model is designed to promote well-learned knowledge transfer over different background conditions for improving robustness and generalizing the model in cross-domain diagnosis tasks so that pump fault diagnosis can be realized under different background conditions. Second, the multi-signal fusion strategy is involved in capturing state information of the pump for establishing the mapping relationship between raw signal and fault pattern by integrating multi-physical signal with weight allocation protocol, thereby implementing fusion processing of multiple types of signals. Finally, a hyperparameter optimization method is explored to align with the transfer learning model by integrating Grid Search with the Gradient Descent algorithm for further improving diagnosis performance and thus achieving optimization of hyperparameter settings. The results show that, compared with other models, the OMTCNN model has excellent performance for pump fault diagnosis and can better realize pump fault diagnosis under different background conditions.


Intelligent composite structures with self-aware functions are preferable for future aircrafts. The real-time location of damaged areas of composites is a key step. In this study, deep transfer learning was used to achieve the real-time location of damaged areas. The sensor network obtained acoustic emission signals from different damaged areas of the aluminum alloy plate. The acoustic emission time-domain signal is transformed into the input image by continuous wavelet transform. The convolutional neural network-based model automatically localized the damaged area by extracting features from the input image. A small amount of composite acoustic emission data was used to fine-tune some network parameters of the basic model through transfer learning. This enabled the model to classify the damaged area of composites. The accuracy of the transfer learning model trained with 900 samples is 96.38%, which is comparable to the accuracy of the model trained directly with 1800 samples; the training time of the former is only 17.68% of that of the latter. The proposed method can be easily adapted to new composite structures using transfer learning and a small dataset, providing a new idea for structural health monitoring. Fiber-reinforced composites have been widely used in the marine, automobile, and aircraft industries due to their excellent mechanical properties ,2. HowevAcoustic emission (AE) is a technique that can extract and characterize material damage characteristics in real time. Elastic waves generated during material damage are detected by AE sensors and converted into electrical signals. Therefore, AE technology can monitor the condition of the structure without external excitation . Many reIn recent years, with the development of data-driven methods, machine learning (ML) methods represented by deep learning (DL) have been applied to damage/defect detection and structural performance evaluation. Sause et al.  used neuUsing a small amount of data to obtain a model with good performance has become a pressing problem for DL-based composite structural health assessment. The damaged area location method proposed in this study has the following advantages: (1) Using the transfer learning strategy, the damaged area location model can be transferred to different structures with a small amount of data, which greatly reduces the cost of building the composite material database. (2) The trained model can be used as an intelligent perception module for vehicles with the composite structure to locate the damaged area in real time. In addition, related studies have shown that reducing the complexity of data sets can potentially improve the accuracy of deep learning models ,32. In tThe contents of this paper are organized as follows: In Aluminum alloy plates and composite laminates of the same size were prepared for AE experiments. The grade of the aluminum alloy plate is 2A12T4 and the size is 500 × 500 × 5 mm. The CFRP composite laminates were prepared by hot pressing plain weave prepregs. The prepreg consists of T300 carbon fiber fabric and epoxy resin (approximately 42% resin volume fraction), with a thickness of approximately 0.25 mm. The prepreg was cut to a predetermined size (500 × 500 mm) and stacked 20 layers at a lay-up angle of 0°, as shown in Researchers have used various types of artificial damage sources  as AE signal sources ,35. The The panels are supported by polymer foam to minimize vibration and energy transfer from the environment to the panels. A cylindrical piezoelectric AE sensor  with a dIn this section, a CNN-based damage area localization model is developed. The powerful feature extraction capability of CNN is applied to learn the nonlinear mapping relationship between CWT images and damage areas to achieve damage location identification. For small training datasets, a transfer learning framework is developed to improve the generalization performance of the model.In order to train the deep learning model, the captured AE signals must be processed in the time–frequency domain. Based on the AE signals obtained by PLB experiments on aluminum alloy plates and composite laminates in Continuous wavelet transform (CWT) is a time–frequency transform method, which is widely used in SHM applications ,37. The CWT The time–frequency scale diagrams of the data from the four sensors are arranged according to the position of the sensors, and the obtained RGB images are used as the input data of the deep learning model, as shown in Convolutional neural networks are a class of deep neural networks most commonly used in visual image recognition. CNN mainly consists of convolutional layers, pooling layers, and fully connected layers. The convolutional layer performs convolutional operations on the input data to extract feature mappings, whereas the shallow convolutional layer extracts basic features, such as edges and contours, and the deep convolutional layer extracts abstract features, such as the entire image. The convolutional layer consists of a set of filters  with learnable weights that perform the main computational tasks in CNN. In a 2D convolutional layer, forward propagation can be expressed as:The pooling layer achieves data reduction by down-sampling. The pooling layer is typically located between two convolutional layers and serves as a regularizer. It reduces both the amount of data passed to the next layer and the amount of computation required. Although the pooling layer loses data information, it prevents overfitting. Max pooling is the commonly used form of pooling and is expressed as:In this study, the VGG16 architecture is used as the damage area detection model. VGG16 is a 2D CNN architecture proposed by Simonyan . Its outA deep learning model typically contains millions of parameters, so a considerable amount of data is needed to effectively train all the parameters to achieve highly accurate predictions. However, in many cases, it is difficult to obtain a large amount of data, which limits the application of deep learning models. A small dataset will also greatly affect the generalization ability of the model. The solution to overcome these problems is called transfer learning , in whicAcoustic emission refers to the phenomenon in which a material locally emits transient elastic waves due to the rapid release of energy. Acoustic emission is also called stress wave emission. The elastic wave emitted by the acoustic emission source propagates through the solid medium to the surface of the object, causing the surface to vibrate mechanically. In solid media, the reflection and refraction of stress waves follow the same physical laws. In addition, due to the difference in the elastic modulus, density, and internal structures of different media, the propagation speed and attenuation of the stress wave are also different. The similar propagation laws and different propagation details of stress waves in different solid media support the rationality and effectiveness of transfer learning. In this study, the VGG16 model is first pre-trained using the AE data of the aluminum alloy plate to obtain the basic model of transfer learning. Next, one part of the network parameters of the pre-trained model is frozen to retain its ability to extract the shallow features of time–frequency image data, while the other part of the network parameters is retrained using the AE data of CFRP laminates to adjust its higher-order feature representation and make it more relevant to the damage area identification task of composite laminates, as shown in The DL model is trained and tested based on the AE dataset of the aluminum alloy plate to obtain a pre-trained model for composite laminate. First, the hyperparameters of the DL model are initialized and the training set is used to train the model parameters. Second, the validation set is used to verify the prediction accuracy of the model, and the hyperparameters of the model are adjusted according to the inference results of the validation set. Finally, when the prediction accuracy meets the requirements, the training is stopped and the model is saved. After 440 iterations, both the loss and accuracy of the model converge, and the accuracy of the model on the validation set is 97.3%, as shown in The performance of the model on the test set is shown in According to the transfer learning framework shown in However, the training cost of the transfer learning model is significantly lower than that of the direct training model. In terms of the amount of data required to train the model, the accuracy rate (96.38%) of the transfer learning model trained with 900 training samples is already comparable to that of the direct training model (96.94%) trained with 1800 samples. This shows that transfer learning can achieve excellent model performance with a small amount of data. When the sample size reaches 1350, the accuracy of the transfer learning model has stabilized at 96.94%, while the accuracy of the direct training model is still rising, which indicates that the sample size of 1350 is not enough for the direct training model. In addition, the accuracy obtained by the TOA method is 73.89%, which is significantly lower than that of the deep learning model. In contrast, in terms of model training time, when the accuracy rate is similar, the training time of the transfer learning model  is only 17.68% of that of the direct training model , as shown in In order to verify the robustness of the transfer learning model, different levels of gaussian noise are added to the time–frequency images in the test set, which are input into the transfer model trained by 1800 samples. As shown in This study proposes a method based on deep CNN and transfer learning for damage area real-time localization of CFRP composite laminates. The deep transfer learning model is well able to identify the location of damaged areas in composite laminates in real time. Instead of manually extracting the discriminative features of the AE signal, this DL-based model can directly localize the damaged area of composite laminates by applying CWT to the original acoustic emission signal. By fine-tuning some parameters of the pre-trained model, the number of model parameters and samples that need to be updated is greatly reduced and the training efficiency of the model is improved. When the training sample size is 900, the prediction accuracy of the transfer learning model is equivalent to that of the model trained directly with 1800 samples. At the same time, the transfer learning model can maintain high recognition accuracy under certain noise and has good robustness. The wrong identification of damage area locations usually occurs near the edge of laminates and sensors. There are two reasons for this result: (1) the uncertainty of the experimental setup, and (2) dispersion and edge reflection during elastic wave propagation. The present study demonstrated that the proposed damage area localization method can be easily adapted to new composite structures and damage types using transfer learning and small sample sizes. This study will provide a new idea for real-time structural health monitoring and the intelligent perception of composite materials.Due to the high cost of composite experiments, the acoustic emission signal from the damaged source can only be simulated by non-destructive experiments, such as the Hsu–Nielsen pencil lead break experiment used in this study. However, this is still different from real composite damage. To overcome these limitations, two future research plans have been established: (1) create a finite element model to simulate the propagation of stress waves caused by impact and acoustic emission signals generated during composite damage, which is desirable to locate the damaged area more efficiently; (2) use deep learning algorithms to identify the relationship between acoustic emission signals and failure modes of composites.


Positional data in team sports is key in evaluating the players’ individual and collective performances. When the sole source of data is a broadcast-like video of the game, an efficient video tracking method is required to generate this data. This article describes a framework that extracts individual soccer player positions on the field. It is based on two main components. As in broadcast-like videos of team sport games, the camera view moves to follow the action and a sport field registration method estimates the homography between the pitch and the frame space. Our method estimates the positions of key points sampled on the pitch thanks to an encoder–decoder architecture. The attention mechanisms of the encoder, based on a vision transformer, captures characteristic pitch features globally in the frames. A multiple person tracker generates tracklets in the frame space by associating, with bipartite matching, the player detections between the current and the previous frames thanks to Intersection-Over-Union and distance criteria. Tracklets are then iteratively merged with appearance criteria thanks to a re-identification model. This model is fine-tuned in a self-supervised way on the player thumbnails of the video sample to specifically recognize the fine identification details of each player. The player positions in the frames projected by the homographies allow the obtaining of the real position of the players on the pitch at every moment of the video. We experimentally evaluate our sport field registration method and our 2D player tracker on public datasets. We demonstrate that they both outperform previous works for most metrics. Our 2D player tracker was also awarded first place at the SoccerNet tracking challenge in 2022 and 2023. With the growing popularity of sports analytics in recent years, there has been recent interest in player positional data. Knowing the position of each player on the field is, indeed, a key to evaluate the players’ individual and collective performances. A solution for outdoor sports is to equip the players with GPS devices. Radio solutions are also available for indoor sports but they require the arenas to be setup with specific receivers. Moreover, the positional data are, most of the time, kept private by the teams. This prevents analysis by the opposing team. Thus, video tracking becomes the sole solution for obtaining these data.Tracking team sport players in videos has been an active research topic. It is a multiple object tracking (MOT) problem with specific challenges. Players on the same team, indeed, share a very similar appearance since they wear the same jerseys. They move quickly and can adopt various poses, which can make their detection difficult. Some play events, such as soccer corners, generate a high number of occlusions. The tactical camera mainly used in broadcast video has a large field of view that allows the visualization of a large part of the pitch. However, the players are small in the images (about 100 pixels high and 50 pixels wide in standard HD videos). The access to fine identification details is consequently limited. The camera view can also move quickly to follow the action. Player detection and tracking are then made complex by the motion blur and the large moves of the players’ bounding boxes. Since, in a broadcast-like video, the tactical view moves, the first component is a soccer field registration method. This determines the position of the soccer pitch in each video frame. The homography between the pitch space and the frame space is computed by estimating the positions of key points sampled over the pitch. One of the primary challenges of the registration stage is to process different partial views of the field, with differing visual characteristics, in order to correctly estimate the homography regardless of the specific view.The second component is a 2D player tracking algorithm. It generates tracklets in the frame space. The major challenge for robust player tracking is to maintain the correct identity of each player over time, despite variability in pose and resolution, frequent occlusions, and temporary exits from the field of view.By projecting the results of the 2D tracking on the pitch space thanks to the estimated homographies, our framework outputs the individual positions of the players on the pitch.In this paper, we present a new framework that can extract the individual players’ positions from a broadcast-like video of a soccer game. Although commercial products  have beeWe propose a new sports field registration method that better captures relevant visual features distributed across the entire image. Our encoder–decoder model is based on a vision transformer and generates heatmaps to locate the pitch keypoints. Previous sports field registration methods use models based on convolutions ,11,12,13We present a robust player tracking approach that implements a low-noise tracklet generation and fusion strategy, and an online learning method for fine-grained re-identification. Our method starts by associating, with bipartite matching, the player detections of successive frames with Intersection-Over-Union (IoU) and distance criteria. The number of player identity switches is reduced by compensating the camera motion between the frames. This quick method generates good quality tracklets between the instances when players occlude each other. For the re-identification (re-ID) of the players, previous methods use a frozen convolutional neural network during the tracking ,16,17,18We demonstrate the performance of the proposed methods through extensive experiments on several datasets. Our homography estimation method is state-of-the-art and outperforms previous works on multiple metrics using the WorldCup  and the Our contributions cover each of the following components:The first section of this article is dedicated to the state-of-the-art in terms of sports field registration and team sport player tracking. In the second section, we describe our methods. The third section is dedicated to the experimental results. We separately demonstrate the efficiency of our sports field registration method and our tracking method. We also combine them to generate individual player trajectories. In the fourth section, we discuss our findings before concluding in the sixth section.In order to accurately determine the positions of players in the field space, it is essential to localize the soccer pitch within the input frame when the camera field of view adjusts to track the action. Previous studies in the literature have addressed the registration of sports fields in videos, with some focusing on utilizing initial annotations ,21,22 anThe initial approaches for registering sport fields use conventional image operators to detect the lines and ellipses on the field, such as Hough transforms  or luminSome recent sports field registration approaches leverage the deep neural network’s ability to generate high-quality segmentation results. The process of these methods can be summed up into three steps. The first step uses a deep neural network to segment the field elements (lines or zone). The second step matches the segmented elements of the input frame with the pitch model elements to generate the homography. Finally, the results are refined in a third step to improve the homography quality.For the segmentation networks, various models have been used: a VGG-16 model  by HomayVarious techniques can match the segmented elements with the pitch model element. Homayounfar et al.  ran a bra et al.  distillea et al.  estimatea et al.  estimatea et al. .Regarding the refinement strategies, Sharna et al.  resort tOther sports field registration methods estimate the homography by directly locating key points without preliminary segmentation. Thus, the frameworks of Jiang et al.  and FaniThe abundance of data is key to training a sports field registration method based on a deep neural network. Many of the previously introduced works resorted to private datasets for training and testing their method. Some authors, however, released their dataset publicly, which makes comparisons easier. For soccer, the WorldCup dataset  is oftenAn efficient player tracking system is required to detect and track the players in all video frames so as to later determine their position on the soccer pitch. Team sport players tracking is a specific sub-task of multiple people tracking or multiple object tracking (MOT). The development of multiple people tracking frameworks has been mainly driven by video surveillance applications. Public challenges were proposed to evaluate the approaches to common datasets. One significant example is the MOT16 challenge . Its traMOT algorithms can be classified into two main categories: online methods and offline methods.Offline methods take advantage of the complete sequence of images. These methods often adopt a graph-based approach ,39, wherTo tackle real-time scenarios, online methods associate new detections with existing tracks using only the current and past frames. To match new detections with existing tracks, the SORT algorithm  uses a KEmploying separate networks for detection and re-ID offers the advantage of addressing the opposite objectives of each task. The detection network focuses on learning shared features to recognize humans, while the re-ID network aims to learn unique features for each individual. However, this approach can introduce scalability challenges, as each detected bounding box requires independent processing by the re-ID network. To mitigate these scalability concerns, single-shot methods have been proposed. These methods utilize a single network to generate both the bounding box coordinates and the corresponding re-ID vectors . Thus, JTraditional multiple object tracking (MOT) methods often result in frequent identity switches when applied to team sport players. Whenever players go out of the field of vision or remain occluded for a significant duration, new identities are assigned to them when they reappear. The varied poses of the players also hinder their detection. Finally, the fast motion of the players and the camera is also a challenge for the prediction of bounding box positions. Consequently, traditional MOT algorithms have difficulties in achieving a good performance on team sports datasets without modifications see .The difficulty of team sport player tracking is that, most of the time, the players from the same teams wear the same jersey. One way to distinguish them is to use jersey numbers. The first jersey number recognition methods, such as the framework of Ye et al. , have beJersey numbers are, nevertheless, not visible from any camera view point. In many cases, appearance details are the only way to re-identify players. Re-identification models have been successfully utilized in the context of basketball. Indeed, the size of the play field makes the players’ thumbnail resolution often important enough to access fine identity traits. The framework of Teket and Yetik  is basedSome MOT frameworks have been specifically designed to track team sport players. Manafifard et al. propose a review article about soccer player tracking . BackgroRecent approaches leverage deep neural networks to detect the players. For example, Theagarajan and Bhanusome  use YOLOGeneric MOT algorithms have been utilized as the base of frameworks targeting sport data. For example, the DeepSORT tracker  is used A great challenge still remains in the disambiguation of the players from the same team, especially when their bounding boxes are very small in the video. The design of efficient re-identification methods that can leverage fine details, visible on the scale of a few pixels, is essential to achieve an overall good player tracking performance.The availability of public team sport player tracking datasets is key to allowing model training and performance comparisons between approaches. The APIDIS dataset  providesThis section describes the three main components of our approach as depicted in Our sports field registration method is built upon the framework of Nie et al. . CompareOur model generates a heat map for each of the K different key points sampled from the court template and a heat map for the background areas.Nie et al. use a ResNet-18  encoder x is a pixel position at the heat map resolution, k is a class (one of the key point or the background), k, x for the class k, and The heat maps generated by our model provide a softmax score for each pixel. This score classifies the pixel to either the key point class or the background class. We use a cross-entropy loss for the training of the model: On ground truth heat maps, each key point corresponds to a disk with a radius of 10 pixels. The aim is to improve the network convergence by regressing large enough areas. t to 1. The ke ypoint position is then computed as the center of mass of the binarized heat map. If this position is inside a border margin of width m, the key point is discarded as its position is most probably not accurate due to missing local context and camera lens distortion.To locate the key points from the estimated heat maps during inference, we binarize the output heat maps by setting all values above a given threshold In the original approach of Nie et al. , the keyith distance between the key points in the court width axis and starting from the camera side using an arithmetic progressionr being the common difference.Consequently, we used our previous work to sample the key points more uniformly in the frame space , in the W being the real-world width of the court and N being the total number of key points on the court width axis. We calculate r by setting a value to Considering the sum of an arithmetic progression, we therefore have the following: The differences between the uniform grid sampling and the perspective-aware grid sampling methods are illustrated in Initializing an encoder with pre-trained weights allows for faster convergence as well as for an improvement in performance in the case where the pre-training domain is larger and close enough to the domain of the task at hand. In recent years, Self-supervised Learning has received a lots of interest due to its ability to learn a good representation from a large source of unlabeled data, more specifically, on transformers ,94. A prWe pre-trained our encoder on the training split of the SoccerNet-v2 dataset  for actiWe used our recently introduced soft contrastive learning pretext task called SCE . In SCE,The dataset is extracted at 2 Fps which creates about 3.3 M images. This large number of images from a domain close to our task of interest provides better-suited initialized weights than an ImageNet supervised pre-trained encoder, as shown in s to 1 and resize them to the original input size. Some random erasing is used to simulate the occlusions of the soccer pitch by the players and encourage the network to use all the visible image features to produce robust results. We replace the pixel three color channel values of several small random areas of the image by 127.5.Data augmentation allows us to virtually increase the number of camera setups and therefore improve the model generalization. Thus, we use horizontal flips to randomly exchange the sides of both teams. We also crop the training images with a random scale factor varying from Previous player tracking methods have a tendency to generate many identity switches when players occlude each other. In order to generate accurate individual trajectories, we designed our method to avoid switches. Unlike previous approaches, where spatial and re-ID criteria are generally used together to match detections to existing tracks, our method employs them in two distinct steps. Our re-ID model is fine-tuned in a self-supervised manner using the player thumbnails from the processed clip, as opposed to the classical approach of using a frozen model. This approach enables us to better recognize the fine identification details of the players.Our 2D player tracking framework follows the tracking by detection paradigm. It is an offline method as it requires the full video sample to optimize player tracks. The first step is to detect the players visible on the field. We use, for this purpose, a state-of-the-art detector: YOLOX . The secThe third step of our method fine-tunes a pretrained re-identification (re-ID) network with a triplet loss formulation so as to later match tracklets from the same player. The network training data are the non-ambiguous player thumbnails from the previously generated tracklets. The fourth step merges the generated tracklets based on the similarity of their re-ID vectors. It also makes sure that a player does not appear twice in a video frame and that he does not teleport. The whole process is summarized in We train a YOLOX model  to detecOur tracklet generation algorithm is online. It is based on the classic SORT approach . The algBeing able to accurately predict the updated position of a track in the current frame is a key factor to correctly match it with the detections of the current frame. Many previous works resort to a Kalman filter  to prediThe camera motion between two frames is estimated by first extracting good features to track  in the pWe want to avoid as much as possible having several identities inside a single tracklet because the generated tracklets are later used to fine-tune the re-identification network. We detect ambiguous bounding boxes by checking if, at a given time, the IoU between two different tracklet bounding boxes exceeds a threshold To generate full player tracks, we need to merge the tracklets from the same players. We carry out this task with appearance criteria. We assume that the players keep the same appearance during the full video sample. Since all the players from the same team wear the same jersey, identifying them from the tactical camera point of view is a hard task. We start with a re-ID model pretrained on the target domain and fine-tune it so as to learn the distinctive appearance details  of the players present in the video clip. This process does not require any annotation. We only use our previously generated tracks as training data.Our re-identification model is a Multiple Granularity Network  with a RIn order to specialize the re-identification model to the target test sequence, the model is fine-tuned a second time for each video sample by using the previously generated tracklets. We use the player thumbnails of our previously generated tracklets as training data. However, to train with clean input data, we discard thumbnails that contain several players. So, when, in a given frame, player bounding boxes are marked as ambiguous see , their rOnce our re-identification model has been fine-tuned, we compute a re-ID vector for each tracklet by taking the mean re-ID vector of We also noticed that, with the way the video samples are filmed, if a player leaves the camera field of view on one side of the frame (left or right), he will most likely re-enter the field of view from the same side. In order to prevent player teleportation, we consider that two tracklets cannot belong to the same player if the distance in the frame space between the last position of the first tracklet and the first position of the second tracklet is above Given our sport field registration method and our player tracker, we can generate the trajectories of each player on the soccer field. Our tracker in the frame space generates bounding boxes and their associated player identities. For each frame, we project the centers of the bounding boxes’ bottom edges to the pitch space thanks to the homography generated using our sport field registration method. We therefore obtain the positions of each player in the pitch space for all video frames.We first evaluate our sports field registration method and player tracker separately before combining the outputs of these components to determine player positions in the pitch space.K is set to 91. The RANSAC re-projection threshold is set to 8 pixels. We experimentally set m to 4 pixels.Our implementation is based on the Pytorch framework. After normalization with the ImageNet mean and standard deviation values, batches of two images at a  dataset  and 100  dataset . The leas is set to 0.7. We also randomly erase 62 rectangles of a size of 45 by 100 pixels in order to mimic the occlusions of the soccer pitch by the players.For data augmentations, the random image flip probability is set to 0.5. The minimum random crop scale factor strong-α and strong-β and symmetrize the loss. The random aspect ratio for random resized crop is sampled between For the self-supervised pre-training of our encoder, we use for SCE parameters, as defined in the original paper , the temThe decoder layers are initialized with default Pytorch uniform distributions.The projection error is the average distance in meters between points randomly sampled in the frame on the visible part of the pitch and projected with the ground truth and the estimated homography;The re-projection error is the average of the distances in pixels, normalized by the frame height, between points randomly sampled on the visible part of the pitch and re-projected with the ground truth and the estimated homography.We use four metrics to evaluate the performance of our approach:We use the publicly available implementation of Chu et al.  to comput to 0.99.The WorldCup dataset  has beenWe report the results obtained by our method in u et al. .t to 0.9975. Compared to three previous works, our framework achieves the best results for all the metrics. Our method reduces the projection error by 27% compared to the method of Chu et al. [We report the results of our method and previous works on the TS-WorldCup dataset  in Tableu et al. . All thet to 0.9975, except for the experience with the Resnet-18 encoder where it is set to 0.99. For each of these ablation experiments, the performance decreases as shown in We also perform ablation studies on the TS-WorldCup dataset to evaluate the performance of our contributions. In a first experiment, we replace the perspective-aware key point sampling using the uniform sampling used by previous works ,11. In aTo visualize the spatial attention from the encoder ViT-tiny, we use the method proposed by reference . It is iOur tracking method is implemented using the Pytorch framework. Our re-identification network is provided by the FastReID project . It takeWe evaluate the performance of our 2D tracker using five commonly used metrics. The Higher-Order Tracking Accuracy (HOTA)  aims at The SoccerNet tracking dataset is composed of 57 soccer clips of 30 s. In the training set and 49 clips in the testing set. The clips are filmed with a single tactical view that follows the action. The annotations provide tracking data for the players, referees, and ball with no distinction of category between the persons and the ball. In our experiments, we therefore handled the ball for the detection, the re-ID, and the tracking tasks in the same way as we did the players and referees.The first part of our experiments uses the oracle detections. In this case, we disable the ambiguous tracklet splits as our tracklet generation algorithm generates very few ID switches with the bounding boxes ground truth as input data. We performed ablation studies to measure the impact of each component on the tracking performance. The results are provided in the first part of We also compared our player tracking algorithm with previous generic trackers ,45,46 anFinally, we evaluated the global tracking performance of our method and previous works by using custom detections. The results are shown in the last part of The SportsMOT dataset  containsTo generate player locations on the pitch, we trained our sport field registration method on the WorldCup training dataset, and an additional 201 images randomly selected from the SoccerNet tracking dataset, in order to be close to the target domain. For the images from the SoccerNet tracking dataset, we manually annotated the homographies.The qualitative results of some clips of the SoccerNet tracking dataset are presented in This experiment utilized a single NVIDIA A100 40 Gb GPU, which was installed in a server equipped with an AMD EPYC 7543 CPU. With our current unoptimized implementation, the total execution time of our framework, for the 30 s length SNMOT-194 sequence from the SoccerNet tracking dataset, is 16 min and 21 s. A significant part of the processing time is spent in the online fine-tuning of the re-ID network (7 min and 25 s).We proposed a new sports field registration method and experimentally demonstrated its performance. Thanks to the usage of a vision transformer architecture, the self-supervised pre-training of our encoder, the perspective-aware key point sampling, and data augmentation techniques, our method outperforms previous works on two public datasets. Furthermore, these contributions have a low impact on the computational complexity in terms of inference time.We also proposed an innovative player tracker in the frame space that first generates non-ambiguous tracklets with spatial criteria and then iteratively merges them according to re-ID criteria. The introduced camera motion compensation helps generate tracklets with fewer identity switches, while the self-supervised fine-tuning of the re-ID network on tracklets in each video clip allows for a better recognition of the fine identification details of the players. Our tracking approach also outperforms previous works on two public datasets and was awarded first place at the SoccerNet tracking challenge in 2022 and 2023. Our method is, however, offline. The fine-tuning of the re-ID network is, by design, computationally intensive. It minimizes the number of identity switches at the cost of longer execution times. Therefore, it currently does not fit scenarios where position results must be delivered in real time.We conducted a qualitative evaluation of our complete framework using soccer video clips. In this context, we generated player trajectories on the soccer pitch and noticed the consistency of the results. A quantitative evaluation will, however, be needed to validate the full system. To accomplish this task, we would require a dataset that combines video clips with ground truth player positions on the pitch. To the best of our knowledge, no such publicly available dataset currently exists. We would like to point out that, although all these experiments were conducted on soccer data, the generic nature of our framework makes it suitable for any team sport video shot with a single moving camera.In this article, we described a new framework for individually locating soccer players from broadcast-like videos. The framework comprises a new sports field registration method and a new player tracker within the frame space. Both components outperform the current state-of-the-art on public datasets. The projection of player positions, individually tracked in the frame space, using the homography obtained through our sport field registration method, enables us to derive player trajectories in the pitch space.As part of our future work, we intend to perform a quantitative evaluation of player positions on the pitch. We also wish to extend our method to identify and localize the players throughout an entire game. An additional challenge in this scenario is managing player substitutions. We believe that recognizing jersey numbers could significantly enhance player identification. This could be approached as a classification problem, where each player thumbnail is assigned labels corresponding to numbers between 0 and 99, and an extra class for cases where the number is not visible. In addition, incorporating the temporal dimension along the tracklets could contribute to enhancing the robustness of jersey numbers recognition. Our iterative algorithm, which currently merges tracklets based on a re-ID criteria, could be extended to incorporate a new prioritized criteria based on jersey numbers. Finally, we would also like to test our approach on more challenging sports, such as rugby, where player occlusions are even more frequent.