With the advancement in big data and cloud computing technology, we have witnessed tremendous developments in applying intelligent techniques in network operation and management. However, learning- and data-based solutions for network operation and maintenance cannot effectively adapt to the dynamic security situation or satisfy administrators’ expectations alone. Anomaly detection of time-series monitoring indicators has been a major challenge for network administrative personnel. Monitored indicators in network operations are characterized by multiple instances with high dimensions and fluctuating time-series features and rely on system resource deployment and business environment variations. Hence, there is a growing consensus that conducting anomaly detection with machine intelligence under the operation and maintenance personnel’s guidance is more effective than solely using learning and modeling. This paper intends to model the anomaly detection task as a Markov Decision Process and adopts the Double Deep Q-Network algorithm to train an anomaly detection agent, in which the multidimensional temporal convolution network is applied as the principal structure of the Q network and the interactive guidance information from the operation and maintenance personnel is introduced into the procedure to facilitate model convergence. Experimental results on the SMD dataset indicate that the proposed modeling and detection method achieves higher precision and recall rates compared to other learning-based methods. Our method achieves model optimization by using human–computer interactions continuously, which guarantees a faster and more consistent model training procedure and convergence. The modern world has been in the midst of significant changes in network-based informatization and digitalization, amid surges in 5G data transmission, cloud data services, and distributed and federated data computing with machine intelligence. In this context, human-oriented network monitoring, operation, and maintenance approaches have become less able to handle current network administration tasks. Traditional IT operation and maintenance teams are confronted with more complicated network infrastructures, essential services, and online business environments. Applying machine intelligence in IT operations strengthens and optimizes the existing operations and maintenance procedures, with machine learning aiding in specific scenarios where problems cannot be easily identified or solved by manual decisions and operations. AI technology is widely believed to extend human analysis and understanding capabilities for network system monitoring and maintenance with high confidence and low cost.System observation data in network operation scenarios have different types and presentation formats, including historical data, traffic data, log data, network device status, resource usage data, etc. To be specific, daily real-time monitoring and detection data of network status are mainly network system indicators, such as resource indicators of servers and network devices, online business operation indicators, and system performance indicators. Essentially, these indicators are usually time-series data, which means they are pairs of timestamps with fixed time intervals and data values of the corresponding indicators. Based on the structural characteristics, these different time-series data can reflect the diversified aspects of the network environment’s operations intuitively and inform us about data storage and processing and transmission stability.It is of great importance to identify the appearance of anomalous time series data during network operation. Network anomaly detection for administration should be able to unveil abnormal fluctuations in system indicators and recognize abnormal network behavior by tracing data variance in time series. Network anomaly detection is an integral part of system monitoring and the underlying core technology of network operations. Any detected abnormal situations may indicate some potential failures in the related services or applications, such as network connection failures, server operation failures, system configuration errors, defective software versions, network resource overloads, server malfunctions, and external attacks.From an operation perspective, it is a great challenge for the network operation personnel to notice abnormal conditions from the massive amounts of monitored data instantly, especially considering the expansion in system scales, the growth in complexity, and the extension of the monitoring coverage. Under these circumstances, building a network operation model and identifying time series data anomalies with machine intelligence are consistently desired by network administrators and service users. This paper proposes to analyze the fundamental characteristics of time-series data with machine learning by designing a new reinforcement learning procedure on network system indicators with human guidance for anomaly detection agents. The network administration personnel are involved in the learning loop by providing interactive recognitions or rebuttals for the currently monitored network indicators and the latest model-detected anomalies. Such input is fed back into the reinforcement learning iterations to guide the model preference to converge toward the human operators’ focus and interest. We believe a practically effective model for network anomaly detection is more valuable if it can detect not only what the training data imply is anomalous, but what the human operators may feel is wrong.We manage to utilize a reinforcement learning method to construct network anomaly detectors, which effectively follows the network status dynamics and benchmarks its time-series consistency with past observations and current human supervisions.We reconstruct the Temporal Convolution Network (TCN) as the principal structure of the Q-network, which better processes the temporal features in network dynamics and achieves a higher accuracy.We introduce human interaction guidance into the learning reward feedback procedure. Users can explicitly provide professional confirmations and evaluations to the currently detected anomalies, which amplify the reward significance for learning model adjustment and convergence.Overall, the presented interaction-guided learning and detection methodology for network anomalies has theoretical and practical significance regarding the design and development of intelligent network operation and maintenance with human operators in the loop. Humans are the observation supervisors and detection evaluators, and the machine learning procedures are the laborers and followers of the human’s dynamic understanding and scenario interpretation of the network situation. We present our methodology with the following innovations and contributions:The rest of the paper elaborates the presented methodology in more detail. Generally, network anomaly detection methods fall into two categories, statistic-based and time-series-modeling-based methods. From a data prospective, there are corresponding processing methods for handling different types of data, such as Time Series Decomposition  and HoltAnomaly detection based on statistics is the earliest and the most widely used method. From data distribution, anomalous data samples may deviate from the statistical distribution of normal samples. First, assuming that the given time series follows a certain probability distribution, if the sample cannot fit the model, it is anomalous. Specifically, statistical methods can also be divided into distance-based, density-based, and clustering-based anomaly detection, such as Local Outlier Factor , K-NeareModeling-based anomaly detection methods for time series first build a prediction model. A residual sequence is obtained by comparing the predicted sequence with the observed sequence and is then analyzed to determine whether there is any inconsistency in the original sequence. Malhotra et al.  use stacAlthough there are various anomaly detection methods for time series, there are not enough benchmarks to test and evaluate their detection performance. Therefore, Lavin et al.  proposedThe above algorithms and tools are general time series anomaly detection methods. Recently, there has been much research efforts and progress on the anomaly detection of time series data in network operation, and they show a trend from methods based on statistics to methods based on machine learning. Xu et al.  proposedGenerally speaking, few methods consider the evolution of anomaly patterns against the observation background over time, which leads to the poor performance of static anomaly detection configurations in dynamic scenarios. An ideal anomaly detection model suitable for intelligent operation and the maintenance environment should have the following characteristics. Firstly, no assumption should be made about the anomaly concept, that is, the definition of the anomaly. What is anomalous should only be derived from the training dataset. Secondly, no predefined threshold should be present, which means that the anomaly detection model should be a logic classifier without fixed thresholds. The detector should have no other manually adjustable parameters except the super parameters, for example, the number of layers in a neural network. Finally, the anomaly detection model should continuously improve dynamically with the accumulation of anomaly detection experience. In other words, the detection model can continuously learn new forms of anomalies and enrich its anomaly knowledge base accordingly.In network management, administrators continuously observe system operation indicators, such as the device resource usage and the link traffic throughput. The characteristics of outliers in such data observations may vary with hardware/software upgrades, system reconfigurations, and the change in service business. It is impossible to maintain the long-term timeliness of outlier detection methods with some fixed thresholds in operation dynamics. As an adaptive control method, reinforcement learning is a viable solution with self-evolution capability. By constructing the anomaly detection process as a Markov Decision Process (MDP), we propose to use Prioritized Experience Replay-based Double Deep Q-Network (PER-DDQN) to evolve anomaly detection models for network anomaly tracing and recognition. This section first presents the core idea and an overview of the MDP and then introduces the anomaly detection and model learning procedure.As shown in (1) State space: (2) Action space: (3) Reward: (4) Strategy: the strategy (5) Discount factor: (6) Value function: Since the reward is only referred to the evaluation of the current state action, it is impossible to measure and compare the advantages and disadvantages of different strategies. Introducing the value function to represent the expected cumulative reward of strategy (7) Optimal strategy: After determining the value function concept, the optimal strategy can be expressed by the following formula. In an MDP setting, the goal is to find the optimal policy The following section details the process of the prioritized experience replay-based DDQN algorithm when evolving the anomaly detection model.According to the ε-greedy strategy, a value is initialized to weigh exploration and exploitation in the initial stage. Meanwhile, it is essential to set the ε-decay value to gradually reduce the probability of random action selection so that the detection agent gradually tends to select the optimal action. This value needs to be determined according to the size of the state space and the length of the time step.To avoid the fluctuation resulting from a single Q-value network, two neural networks with the same parameters are initialized . The priIn addition, a fixed-size experience pool needs to be initialized. The experience pool is designed to ensure the independence of samples in training. In contrast, if continuous samples are used for random gradient descent training, then there will be a strong correlation among the samples and the variance of parameter updating is large, which makes it difficult to converge. As mentioned above, the experience pool adopts sampling with replacement; another advantage of this method is that the samples can be reused to improve data utilization.Nevertheless, the benefit of each network observation sample in the experience pool for model updating differs greatly. In network anomaly detection, owing to the sparse positive samples, it is far more important for the agent to learn from the positive samples. In this case, the priority of the sample and sample can be further defined according to the priority . GeneralThe agent explores the state space by selecting random actions or selects the best actions based on the current strategy. Then, the agent executes an action, that is, it makes a judgment whether the observed time-series data in the network environment are normal or abnormal. A human trainer gives the corresponding reward to the agent after receiving the alarm information (agent action decision). Meanwhile, the agent receives the state of the next time step and stores When the accumulated samples in the experience pool reach a certain amount, mini-batch-sized samples are collected according to the priority to update the Q-network. The probability of each sample being extracted is After adding the weight of correction deviation, the loss function is defined as follows.The overestimation of the According to the mean square error function, the parameters of the Q-network are updated by backpropagation. Then, we use the updated network to recalculate the TD error to update the sample priority of the experience pool. If the number of iteration rounds reaches A TCN is a time series modeling framework based on a convolutional neural network . CompareA TCN uses causal convolution to ensure that the output value does not depend on future information. When calculating the output at The size of the receptive field depends on the network depth, filter size, and dilated factor. In some cases, it is still necessary to increase the network depth to obtain enough receptive field. A residual connection is an effective way to train a deep network, making the network transmit information in a cross-layer way. The residual block contains a series of transformations for the input, adds its output to the input, and activates it.It is feasible for the agent to interact with the environment autonomously based on the reinforcement learning algorithm. However, two principal issues exist in such a process. For one thing, under the premise of computing hardware limitations, the sampling efficiency of the reinforcement learning algorithm is relatively low, leading to a slow convergence speed. Additionally, the deviation in the final concept between model construction and the requirements of operation personnel may result in false alarms and omissions. Therefore, a more effective strategy is to speed up the model’s convergence process under guidance from external operators in a specific state.In this section, we propose an anomaly detection framework for network environment management with interactive guidance from human operators. As demonstrated in The network operation environment represents the whole network system’s dynamic running, including the hardware, software, and data resources supporting normal network services. At each time step, the network operation environment sends the real-time monitoring indicators to the anomaly detection agent as state vectors and displays the network status to the operation personnel through the network operation monitoring system. Simultaneously, the network operation environment feeds back according to key performance indicators, such as response time, throughput, and concurrency.Operation personnel are the decision makers and executors of the network operation system. They can ensure the network’s proper functioning by monitoring the network environment and handling the faults in a timely manner. At each time step, the operation personnel can view the time sequence monitoring indicator in the real-time network and the anomaly detection results returned by the agent through the user interface. Operation personnel evaluate the accuracy of alarm information based on historical experience and professional knowledge to confirm or cancel the alarm. If necessary, they also need to carry out fault handling. Operation behaviors are converted into feedback signals to guide the anomaly detection agent to perform specific actions in different states.Generally, if the agent produces false alarms or underreporting, the operation personnel should give negative feedback signals. If the network status outliers are correctly identified, operators signal positive feedback. Considering the human cost, the interaction between the operation personnel and the agent can be considered an occasional process. In particular, when the operation personnel fail to give immediate feedback, the agent’s anomaly detection results are accepted by default.The anomaly detection agent is an effective tool to detect outliers from the time dimension. In the proposed framework, the interaction is the process in which the operation personnel guide the agent to learn a target strategy. During this process, the operation personnel deliver this strategy through interaction, where in parallel, the agent learns and integrates it from the feedback.In this framework, human–computer interaction is a two-orientation process, which can be accessed through the user interface in the network operation monitoring system. In terms of the agent’s standpoint, human–computer interaction can be divided into active interaction and passive interaction. Taking the agent as the starting point, active interaction outputs the execution results or suggestions to the operation personnel without instructions. In this case, the agent can extract the time series data and analyze the data to make decisions. In terms of a malfunction, the agent sends an alarm to the operation personnel. However, it is common that humans make mistakes, such as failing to confirm the alarm in a timely manner or mistakenly marking historical records. Under these circumstances, the agent is required to be equipped with the correction ability to assist the judgement of operation personnel. To provide more reliable data support, agents tend to compare multiple similar sequences based on historical records and select the most tagged category as the corrected result.Passive interaction refers to the process in which operation personnel intervene to interact with the agent to maintain the network environment. In some cases (service adjustment or system update), the temporal characteristics of network status indicators change significantly. Thus, the agent cannot effectively detect and signal abnormal conditions, resulting in a large number of false alarms and underreporting. Based on the observation and collection of real-time monitoring information, the operation personnel can label historical records to correct the missing and false alarms. Additionally, operation personnel can set some specific model parameters and rules for specific service scenarios and performance requirements, such as the sliding window size, interval step, the setting of the reward function, the size of the weight factor, the alarm setting level, etc. Operation personnel directly adjust the model effect through these user interfaces.In the proposed anomaly detection model, the reinforcement learning agent receives two rewards from different sources, namely, the reward based on the fluctuation of key performance indicators and the reward based on historical experience and professional knowledge. According to their different attributes, the former is defined as an objective reward In the above formula, The empirical reward refers to subjective judgment. How to give it a clear reward mapping to interactive guidance is of great importance. Existing algorithms assume that the reward merely depends on the agent’s action choice. This paper proposes that it also depends on the agent’s current strategy. The reward or punishment after an action selection depends on agent’s action accuracy. It can be explained that the operation personnel tend to give more positive feedback when the strategy of the agent is significantly improved, and when the agent makes mistakes in the near-optimal and relatively stable strategy, it will impose greater punishment on the agent. In this paper, we propose a reward mechanism based on strategy evaluation. The central issue is to give the corresponding reward according to the current strategy evaluation of the agent.The pivotal part is how to describe the performance compared with that under the current strategy. According to the description of the MDP in the previous chapter, the state value function is equivalent to the long-term cumulative reward expectation in the current state, while the action–value function refers to the evaluation of action selection in the current state. The difference between the two values just reflects the comparative advantage of action selection based on the current strategy. Therefore, as shown in Equation (10), this paper uses the advantage function as the evaluation metric of strategy, and it is also the mapping of empirical rewards.Generally, the state value function can be approximately calculated in the reinforcement learning algorithm based on strategy gradient, but the reinforcement learning algorithm based on the value function used in this paper cannot directly obtain In this paper, a positive sample refers to an anomaly, and a negative sample refers to a normal state. Accordingly, True Positive (TP) indicates the number of correctly detected anomalies. False Positive (FP) indicates the number of incorrectly classified normal data. True Negative (TN) indicates the number of correctly detected normal data. False Negative (FN) indicates the number of anomalies wrongly classified as normal.The calculation of the final advantage function is shown in The interaction module is designed to provide human feedback as a reward to the detection agent, so that the model being trained is guided to converge faster according to the operation personnel’s preference. There are inevitably some human errors in network operation and abnormal records are labeled incorrectly. Therefore, the anomaly detection agent should also have a specific mechanism to distinguish and correct the behavior of operation personnel. In this paper, we utilize the Dynamic Time Warping (DTW) algorithm  to handlWith two sequences, The optimal path between The DTW algorithm uses the dynamic programming method to align the two sequences to obtain the regular path. However, it takes a lot of time to traverse all possible paths. In order to improve the efficiency, the number of possible regular paths must be limited.In this paper, ambiguous sequences are recorded as target sequences and historical sequences are recorded as matching sequences. To further reduce the computational overhead, the length of the matching sequence is equal to the length of the target sequence, that is, This section evaluates the performance of the proposed anomaly detection method. We first introduce the datasets, metrics, and settings. Then, the performance of the proposed model compared to the current state-of-the-art schemes for anomaly detection is demonstrated. Finally, we present the effectiveness of the interaction-based guidance mechanism. The experiments are implemented using an i5-9400F CPU @ 2.90 GHz with 16 GB of RAM on Python 3.7 and TensorFlow 2.1.We conducted our experiments on one-dimensional Key Performance Indicators (KPI) and multidimensional Server Machine Dataset (SMD). We list the summary statistics for the datasets in The detailed information of the datasets is described in the following.The KPI dataset is from NetMan Laboratory . The datThe SMD is a 5 week long dataset that was collected from a large internet company. This dataset contains three groups of entities and is made up of data from 28 different machines. We conduct our experiments on the <machine-1-1> dataset. It was divided into two subsets of equal size; the first half is the training set and the second half test set. Domain experts have labeled anomalies and their anomalous dimensions in the SMD testing set based on incident reports. The data collection time interval is 1 min. As shown in The following metrics were applied to evaluate the performance of the proposed approach.In Equation (16), accuracy represents the proportion of samples correctly classified.In Equation (17), precision refers to the proportion of correctly classified anomalies relative to all samples classified as an anomaly. The higher the precision, the lower the false alarm rate.In anomaly detection, recall is the most crucial evaluation indicator, representing the proportion of anomalies detected to all abnormal data (Equation (18)). A higher recall comes with a lower missing report rate.One of the most common anomaly detection issues is that it is hard to achieve a high precision and recall simultaneously. In this case, the F1-score measures the harmonic mean of precision and recall, which can comprehensively reflect the performance of the anomaly detection model.We can adjust the model’s detection attention to the historical observations by tuning the discount factor The length of the input vector of the Q-network was set to 60. The dilations of each layer in TCN were set to In the interaction process, the operation personnel are modeled with multiple attributes as simulation parameters. Specifically, interaction frequency, persistence, and accuracy are used to represent the interaction behavior of the operation personnel.The interaction frequency refers to the proportion of data that operation personnel interact with among all the samples. In the process of active interaction, most of the alarm information contains a continuous number of abnormal samples, and the operation personnel can confirm or negate them in batch, where the interaction frequency is 100%. In the process of passive interaction, that is, when checking and tagging the historical records, the operation personnel have to deal with most of the normal time-series data, in which the abnormal points are very sparse, so the interaction frequency is 0.1%.Interaction persistence refers to the duration steps of operation personnel interaction. Considering that the operation personnel cannot give the agent feedback information continuously due to various objective conditions, the duration steps are generally set to 5. The experience replay mechanism is used in the anomaly detection model; thus, the attribute will not affect the detection performance of the agent.Interaction accuracy refers to the correctness or quality of the feedback information from the operation personnel. As mentioned previously, during the instant monitoring and management operations, administration personnel may inevitably make some mistakes. Such false information may lead to some deviation of the agent’s understanding of the anomaly concept. In this paper, the proportion of operation personnel’s misjudgments in all the interaction samples is simulated with a setting of 0.1%. These samples are used to verify the active correction mechanism based on DTW in the experiment, and the approximate distance threshold is set to 5.In To demonstrate the effectiveness of the proposed method, we first compared it with four state-of-the-art approaches: Donut 18, Buzz 19, Period 20, and OmniAnomaly 21. Next, we used different neural network structures as the Q-network, including a DNN, LSTM, GRU, and a TCN. In this part, we compared the non-interactive model and the interactive model. For the convenience of analysis and display, we conducted a contrasting experiment on the KPI dataset. As shown in In the experiment, based on the 0.001 misjudgment probability of operation personnel, nine misjudged samples were generated. These samples are all underreported samples. Abnormal points are recorded as normal data by operation personnel. For all the misjudged samples, the agent will use the active correction mechanism based on DTW to proofread. This paper presents a robust anomaly detection method based on reinforcement learning and an interaction mechanism in network operation, particularly for monitoring indicators. By combining prioritized experience replay and the DDQN algorithm, reinforcement learning is applied to network anomaly detection tasks in parallel with network operations. To achieve better results on time series data, we propose to incorporate a TCN as the principal structure of the Q-network. Additionally, we further design an anomaly detection framework with interactive guidance from operation personnel. We give clear definitions for the concepts of active interaction and passive interaction during the interaction process. In particular, the combination of objective and empirical rewards and a reward mechanism based on strategy evaluation are proposed for the agent’s internal update. We adopt the DTW active correction mechanism to correct false guidance information produced by the operation personnel. Experimental results on the KPI and SMD datasets demonstrate that the proposed method outperformed other anomaly detection techniques and the interaction mechanism can effectively accelerate the convergence speed of the model.While the current system architecture and detection procedure are complete, there are still some limitations in the proposed approach, which we are investigating further to make effective improvements. First, we need a more effective way to condense the hyper-dimensional network traffic time-series data to present the current network situation to the administration personnel. For instant monitoring, humans can only percept and understand traffic data with proper sampling and forensic representations. Secondly, current human–model interactions are modeled as data labeling with uncertainty. Although we have considered a scenario with human error, the validation and correction mechanisms require more precision and efficiency. Additionally, thirdly, adapting the proposed mechanism with existing online traffic surveillance and detection systems such as IDS/ADS is desirable, especially when the traffic volume in a realistic system could be too much to completely undergo the current modeling and detection procedure. For the data analysis and processing parts, we plan to gain further insights into the method optimization of time series classification so that we can design different anomaly detection methods for diversified service requirements. Furthermore, we expect to build up more in-depth knowledge about the interaction mechanism to probe effective methods for human–agent interactions. While this presented work shows promise for effective combination of computer intelligence and human guidance, we look forward to more sustainable, adaptable, and systematic solutions for applying such mechanisms in realistic network traffic detection and analysis in the near future.


Global motion greatly increases the number of false alarms for object detection in video sequences against dynamic backgrounds. Therefore, before detecting the target in the dynamic background, it is necessary to estimate and compensate the global motion to eliminate the influence of the global motion. In this paper, we use the SURF (speeded up robust features) algorithm combined with the MSAC (M-Estimate Sample Consensus) algorithm to process the video. The global motion of a video sequence is estimated according to the feature point matching pairs of adjacent frames of the video sequence and the global motion parameters of the video sequence under the dynamic background. On this basis, we propose an inverse transformation model of affine transformation, which acts on each adjacent frame of the video sequence in turn. The model compensates the global motion, and outputs a video sequence after global motion compensation from a specific view for object detection. Experimental results show that the algorithm proposed in this paper can accurately perform motion compensation on video sequences containing complex global motion, and the compensated video sequences achieve higher peak signal-to-noise ratio and better visual effects. At present, intelligent analysis technology has realized the detection, identification, tracking, and human behavior analysis of moving targets. It is widely used in the military, intelligent transportation, medicine, and other fields ,2,3. AmoThe motion of the background is usually caused by the change in the position of the recording device, which is called global motion ,9. The mTarget detection in the static background mainly includes the frame difference method , backgroMoving object detection under a dynamic background ,17 mainlThe motion vector between images is obtained first, and then the motion parameters are obtained by fitting the motion model . In pracThe SIFT algorithm has scale invariance and can detect a large number of key points in the image for fast matching. However, since the SIFT algorithm does not consider the geometric constraints of the space, it leads to a high mismatch rate ,27. ORB Although ORB takes less time than SURF, it provides lower matching rates in rotation and shearing scenarios of different strengths. Therefore, SURF is considered a suitable compromise between speed and performance. After the feature point matching pair of the image is obtained by the SURF algorithm, it is necessary to use the MSAC (M-Estimate Sample Consensus) algorithm  to furthThe traditional global motion compensation method is to use the adjacent two-frame images to conduct a global motion estimation to obtain the motion model. By performing model transformation on the previous frame image to predict the next frame image, a compensated frame image is obtained ,35. WhenC. Song et al.  introducIn this paper, we refer to the traditional idea of global compensation and make improvements. After the affine transformation model is obtained by performing global motion estimation on the adjacent front and rear frame images, an inverse transformation model of the affine transformation is established to act on the latter frame image. And sequentially output compensated images of consistent size with a specific view. The video sequence composed of compensated images can directly perform frame difference operations to extract moving targets.We aim at the problem of the high false alarm rate of moving target detection in video sequences caused by global motion and start with the conversion of dynamic background to static background in video sequences. We propose a global motion compensation algorithm based on an affine inverse transform model: By combining the SURF algorithm with MSAC processing, we obtain an affine transformation matrix representing the global motion of the video sequence. On this basis, an inverse transformation model of affine transformation is proposed, which is used to compensate for the global motion. We also realize the conversion from a dynamic background to a static background in a video sequence. Finally, the moving target is detected by the frame difference method.V under the dynamic background is regarded as an ordered set composed of n frames of images:The video sequence kth frame; kth frame; kth frame. Assume that there are m moving targets in total, among which, i.In the Equation , Ik(θk) The parameters for Reflection is the mirror image of the original object. In other words, it can say that it is a rotation operation with 180°. In reflection transformation, the size of the object does not change. A transformation that slants the shape of an object is called the shear transformation. There are two shear transformations X-Shear and Y-Shear. One shifts X coordinates values and the other shifts Y coordinate values. In both cases, only one coordinate changes and the other preserves its values. However, transformations are often composite rather than existing in isolation. Composite transformations can be achieved by combining transformation matrices, as shown in Equation :(2)[T][Xkth frame, that is, the global motion state. The motion parameter kth frame image. This motion state superimposes the current background motion state In the Equation , θkb repDue to the superposition between the global motion and the motion of the moving object, it is difficult to distinguish between the moving object and the background, and the detection of the moving object is more difficult. In order to accurately and completely extract the moving target in the video sequence, it is necessary to remove the influence of the global motion on the target motion as much as possible. Then, the global motion compensation problem of video sequences under dynamic background can be described as follows: Find a suitable algorithm to obtain the parameters of the global motion estimator kth frame. That is the estimation of the global motion state of the image in the kth frame. The motion parameter kth frame image after compensation.In the Equation , θ^kb reSURF is a commonly used feature point matching algorithm. When matching two images in the same scene, we first use the Hessian matrix to generate all the points of interest in the image. Then, we extract the feature points after eliminating the unreasonable interest points, and generate their descriptors. The matching pairs of feature points are obtained through the comparison of descriptors.H is established at a certain pixel in the grayscale image kth frame image:The Hessian matrix is a square matrix composed of the second-order partial derivatives of a multivariate function, which describes the gray gradient changes in all directions. It generates points of interest by obtaining all “suspicious” extremum points. Before constructing the Hessian matrix, the image needs to be Gaussian filtered, and the Gaussian blur coefficient is P and the second-order Gaussian template In the Equation , Luu::LuuIn which:It can be seen that the determinant of the Hessian matrix of each pixel is as follows:Equation  is also O to eliminate non-extremum points.In order to obtain the feature points that can be used for feature matching in As shown in  wavelet  is used V under the dynamic background are respectively established: E is the number of feature points of F is the number of feature points of According to the above method, the feature point sets of two adjacent frames of images And the descriptors in the sets p th feature point in the feature point description subset z th dimension feature of the descriptor. q th feature point in the feature point description subset zth dimension feature of the descriptor.Among Equations –14), R, All feature point matching pairs that meet the above conditions are composed of feature point matching pairs between images V with dynamic backgrounds, and to estimate the background motion, it is necessary to eliminate a portion of feature point matches in To obtain accurate motion parameters between adjacent frame images nsensus)  that oveThe majority of feature point matches in the set 1. Randomly select 2. For the remaining feature points in C of the current inlier set 3. If the cost function k, terminate the process. Otherwise, increment the iteration count and repeat the above steps. The value of the iteration count G is determined by Equation G=lEquation .(17)C=∑Equation .(18)LMk=In the Equation , θbThe global motion compensation, using the inverse transformation model of the affine transformation, is applied to In the sequence The adjacent frames of the video sequence Equation . By compIn the Equation , Ik+1′. Each MTU consists of three main components: multi-task detail network In the motion compensation network For deblurring network The total complexity of the RVDMC algorithm is:As can be seen from Equations  and 31)31), the In this section, we conducted background motion compensation experiments on two video sequences, In experiments, the proposed algorithm is used to estimate and compensate for the motion of The video sequences In The affine transformation model consists of six motion parameters, as shown in Equation . Table 1According to the affine transformation model, For all the consecutive frame images in the video sequences In After estimating the global motion, we apply the proposed background motion compensation algorithm to the video sequences Equation . FinallyThe video sequences From Based on the obtained video sequences By comparing the two frame difference images, significant differences in the object detection results can be observed. In In comparison to the In this paper, we adopt an objective quality metric called Peak Signal to Noise Ratio (PSNR) to demonstrate the effectiveness of the proposed algorithm. A higher PSNR value indicates lower image distortion. We calculate the PSNR values between the grayscale frames of adjacent frames in the dynamic background video sequences From the PSNR value variations in The video sequence Shao et al.  designedThis paper proposes a global motion compensation algorithm for video object detection based on the affine inverse transformation model. The algorithm utilizes SURF in combination with the MSAC algorithm to obtain global motion parameters and further fits an affine transformation model. It estimates the background motion of the entire video sequence. Based on this, an inverse transformation model of affine transformation between adjacent frames is proposed. It is applied to globally compensate for the motion in the entire video sequence. The compensated frames are then outputted in a view created based on the first frame of the video sequence as a reference, completing the transformation from dynamic background to static background.The effectiveness of the proposed algorithm is further demonstrated by evaluating the detected motion objects using frame differencing between adjacent frames and comparing the peak signal-to-noise ratio (PSNR) among different algorithms. However, due to the complexity and non-periodicity of background motion, the proposed algorithm may still introduce some noise in the detected objects. Further optimization and improvements are needed to address this issue.https://youtu.be/hMFII2rpc4s (accessed on 5 May 2023) and https://youtu.be/NAPBZkvBahg (accessed on 5 May 2023).The video processing results can be viewed at


The clinical development of new cancer drugs is based on clinical trials that compare the new drug with the one representing the standard of care. However, it takes some time for the new drug to become available for patient use. In the meantime, the standard of care may have been updated and therefore the efficacy of the new drug needs to be tested against the new standard of care. This objective is usually pursued by performing indirect comparisons: since no “real” trial is available comparing the new drug with the new standard of care, this comparison is carried out by means of a simulated trial wherein patients of real trials are reconstructed individually, pooled together according to an original design of comparison, and finally subjected to standard statistics. Previously published trials are used to perform this comparison. We describe a new method to perform these indirect comparisons, called the IPDfromKM-Shiny method, that lies midway between simple interpolation and advanced statistics. More than 10 investigations have already been conducted in which this method has been used to perform indirect comparisons, especially in the area of haemato-oncology. In the light of these preliminary experiences, the IPDfromKM-Shiny method is proving to be a valid advancement for conducting comparative research in the area of evidence-based medicine.In the area of evidence-based medicine, the IPDfromKM-Shiny method is an innovative method of survival analysis, midway between artificial intelligence and advanced statistics. Its main characteristic is that an original software investigates the Kaplan-Meier graphs of trials so that individual-patient data are reconstructed. These reconstructed patients represent a new form of original clinical material. The typical objective of investigations based on this method is to analyze the available evidence, especially in oncology, to perform indirect comparisons, and determine the place in therapy of individual agents. This review examined the most recent applications of the IPDfromKM-Shiny method, in which a new web-based software—published in 2021—was used. Reported here are 14 analyses, mostly focused on oncological treatments. Indirect comparisons were based on overall survival or progression free survival. Each of these analyses provided original information to compare treatments with one another and select the most appropriate depending on patient characteristics. These analyses can also be useful to assess equivalence from a regulatory viewpoint. All investigations stressed the importance of heterogeneity to better interpret the evidence generated by IPDfromKM-Shiny investigations. In conclusion, these investigations showed that the reconstruction of individual patient data through this online tool is a promising new method for analyzing trials based on survival endpoints. This new approach deserves further investigation, particularly in the area of indirect comparisons. In July 2021, three statisticians of the MD Anderson Cancer Center in Houston published a new method [The main applications of the Shiny method in the field of evidence-based medicine (EBM) have been reported beginning from 2022. Indirect comparisons of efficacy across innovative treatments represent the typical investigation of researchers who employ this approach. The Shiny method, in fact, allows simulated comparative trials to be easily performed to evaluate treatments for which no real head-to-head trial has been conducted.The development of a new drug in haemato-oncology aims at the demonstration of superiority, in terms of overall survival (OS) or progression-free survival (PFS), of the new therapeutic option versus the standard of care (SOC) ,2. When The aim of this paper is to describe the application of the Shiny method for comparative assessment of the efficacy of novel treatments, with special emphasis on haemato-oncological agents.This series of 14 experiences of application of the Shiny method was identified through a standard PubMed search  AND (IPDfromKM OR Shiny)>>) combined with manual Google queries. Eligible papers were 133 from PubMed and 90 from other sources. After examination of the full text of these articles, 14 were found to report a Shiny method analysis. The IPDfromKM software is freely available on the Internet and does not require any login to run. There is also an R-code version that can be requested from the authors  at the University of Houston (see Ref. ). It shoThis analysis , which rThis analysis, published in October 2022 , represep = 0.0014). The investigation by Messori et al. [Relapsed-refractory multiple myeloma (RRMM) is a hematologic malignancy in which several therapeutic innovations have become available in recent years. Two Shiny method analyses ,6 focuseNumerous Shiny method investigations have focused on NHL, especially relapsed-refractory diffuse large B-cell lymphoma. The paper by Messori et al.  analyzedThe Shiny method analysis by Messori and Caccese  comparedIn 2022, a Shiny method analysis  investigThis example is interesting because it shows that the easy indirect comparisons generated by the Shiny method can help to correctly interpret the survival data and permit the standard of care used for comparison to be kept updated. On the other hand, the superiority of relatlimab plus nivolumab vs. nivolumab monotherapy in Tawbi’s trial can be explained because the controls of Tawbi’s trial had more favorable prognostic characteristics than the controls of Wolchok’s trial that proposed nivolumab as standard of care B.In a Shiny method analysis , the effThis study  represenThe Shiny method analysis on first-line immunotherapies in triple-negative in breast cancer  was publMaintenance therapies for patients with advanced ovarian cancer who responded to platinum therapy were examined by Cancanelli et al.  with an This analysis illustrates how the Shiny method can be applied in settings other than haemato-oncology. The Shiny method analysis by Trippoli et al.  re-examiAs previously discussed, one of the main critical aspects of the Shiny method is represented by a biased selection of the studies included in the analysis. Hence, in all Shiny method analyses it is essential to carefully evaluate whether the criteria used to enroll the patients were homogenous among the trials. Despite this, trials that apparently enrolled patients with similar inclusion/exclusion criteria can be affected all the same by a certain degree of heterogeneity in patient selection, and this unnoticed heterogeneity can impact efficacy. To address this problem, a heterogeneity assessment on control groups is suggested for all Shiny analyses . In factTo discuss this issue more extensively, two data sets ,14 deserIn the first case , the deDemonstrating therapeutic equivalence among medicines is the last application of the Shiny method  discusseIn recent years, different researchers worldwide have recognized the scientific validity of the Shiny method  and haveAlthough no large-scale studies have specifically investigated the precision of the method, anecdotical information agrees that the method provides excellent reconstructions of the Kaplan-Meier curves. On the basis of the trials on CLL examined in The reconstruction of individual patient data (IPD) to perform comparative studies is an innovative area of clinical research in which indirect comparisons represent the most typical objective. The 14 experiences reviewed in this article indicate that this type of research is nowadays supported by a sound methodology. However, one should keep in mind that these indirect comparisons, which by definition have a retrospective design, are unable to account for some “between-trial” differences in enrolled patients that can have an important impact on outcomes. To minimize this bias, before conducting a Shiny method analysis, a careful assessment is needed about the inclusion criteria adopted in the different trials. Furthermore, after finishing a Shiny method analysis, post-hoc statistical assessments are always needed to explore the degree of heterogeneity affecting control groups of included trials. Further and more specific tests still need to be developed to improve the performance of these post-hoc assessments of heterogeneity. All in all, this review confirms that the main strength of the Shiny method approach lies in the communication strength of its typical multi-treatment graphs, which represent a new and easily interpretable tool in the armamentarium of evidence-based medicine.


Fetal magnetocardiography (fMCG) has proven to be an important tool for the prenatal monitoring of electrical cardiac activity; however, the high cost of superconducting quantum instrumentation (SQUID) poses a limitation for the dissemination of fMCG as a routine clinical technique. Recently, optically pumped magnetometers (OPMs) operating within person-sized, cylindrical shields have made fMCG more practical, but environmental magnetic interference entering through the shield opening substantially degrades the quality of fMCG signals. The goal of this study was to further attenuate these interferences by placing the OPM array within a small ferrite shield. FMCG recordings were made with and without the ferrite shield in ten subjects inside a person-sized, three-layer mu-metal cylindrical shield. Although the fetal signal was slightly attenuated, the environmental interference was reduced substantially, and maternal interference was also diminished. This increased the signal-to-noise ratio significantly and improved the resolution of the smaller waveform components. The performance improvement was highest in the axial direction and compensated for a major weakness of open-ended, person-sized shields. The ferrite shield is especially beneficial for the deployment of triaxial OPM sensors, which require effective shielding in all directions. Fetal magnetocardiography (fMCG) is a noninvasive modality that records the magnetic activity of the fetal heart during pregnancy. It has emerged as an important complementary diagnostic tool to fetal echocardiography, which records mechanical activity and blood flow, and an alternative to fetal electrocardiography, which records electrical activity but is not used clinically due to its poor signal quality. Numerous studies have shown that fMCG can be recorded reliably throughout the last half of pregnancy and provides valuable information, including assessments of fetal heart rate, beat-to-beat fetal heart rate variability, fetal activity, and fetal rhythm. Clinically, the most important application of fMCG is the diagnosis of fetal arrhythmia ,2. The uRecently, fMCG technology has been revolutionized by the emergence of optically pumped magnetometers (OPM) ,5,6, a cSeveral methods have been investigated to improve the shielding performance of open-ended, person-sized shields. It is well known from simulations that lengthening the shield improves performance; however, this increases the cost and difficulty of finding suitable space in a clinical setting. Adding a fourth layer of shielding to a typical three-layer, person-sized shield can result in performance similar to that of a large-size MSR , but agaThe problem of leakage interference from the shield opening is considerably more severe for fMCG than for other applications. For magnetoencephalography (MEG) and even MCG, the sensors can be positioned much closer to the closed than the open end of the shield. For fMCG, however, the sensors are positioned near the middle of the shield. The leakage interference is especially strong in the longitudinal direction because, for cylindrical geometries, the shielding factor is approximately ten times higher in the transverse direction than in the longitudinal. In the study by Strand and coworkers, the sensors were oriented to record only the transverse components of the fMCG signal due to the low signal-to-noise ratio (SNR) of the longitudinal component .In this study, we investigated the use of a small ferrite shield to augment the performance of a conventional person-sized magnetic shield. Small ferrite shields have been used previously by Kornack and coworkers to circumvent the Johnson noise associated with mu-metal shields . To our The measurements were performed within a cylindrical three-layer mu-metal shield  with an inner diameter of 75 cm and length of 2.5 m A. One enThe study took place at the Wisconsin Institute for Medical Research at the University of Wisconsin–Madison, where the earth’s magnetic field is ≅ 54 μT. The residual field was reduced to approximately 10 nT by placing the shield perpendicular to the earth’s magnetic field. A ferrite cylindrical shield , depicted in 1/2. Each sensor measures two orthogonal components of the magnetic field, one parallel to the long axis of the sensor (Z) and the other parallel to the short axis (Y). The Gen2 sensors were smaller than the Gen1 sensors. The Gen3 sensors became available toward the end of the study. They are triaxial sensors  and have a slightly lower magnetic field resolution. A 3D-printed plastic holder accommodated 11 OPM sensors arranged in an offset square grid pattern of 9 × 9 cm —were used to record the fMCG. The Gen1 and Gen2 sensors were dual-axis sensors with an intrinsic magnetic field resolution of 10–15 fT/(Hz)9 × 9 cm C. The se9 × 9 cm D, and thThe vertical placement of the sensor array within the ferrite shield affects the shielding effectiveness and signal fidelity. If the sensors are positioned well below the top of the shield, the interference will be strongly attenuated but the signal may also be diminished. If the sensors are positioned near the top of the shield, the signal loss will be minimal, but the shielding effectiveness will also be reduced. To help determine an optimal placement, we measured the shielding factor as a function of the vertical position of the sensor array with respect to the top of the ferrite shield, using an external magnetic coil to simulate environmental interference. The shielding factor is defined as the ratio of the magnetic field in the absence of the ferrite shield to the field in the presence of the shield.The effectiveness of the ferrite shield in attenuating environmental interference was assessed by placing the center of the OPM array 140 cm from the shield opening and recording the interference with and without the ferrite shield. Five dual-axis and four tri-axis OPMs were used to obtain the noise in all three directions. Each measurement lasted 30 s. The power spectra for each direction in both cases were obtained using Welch’s method in Matlab.The protocol was approved by the UW-Madison Health Sciences IRB, and informed consent was obtained from all subjects. The subjects were 10 adult pregnant women, studied at 21–36 weeks of gestation. Seven of the pregnancies were uncomplicated. One was complicated by fetal long QT syndrome (LQTS). A second was complicated by endocardial fibroelastosis and fetal AV block associated with isoimmune disease. A third was a case of congenital heart disease, tetralogy of Fallot.The data were collected with the mother lying prone on two sections of foam mattress separated by a gap. The sensor array was covered by a thin piece of foam and placed within the gap in contact with the mother’s abdomen from below. Airflow was applied to the sensors to prevent them from overheating. After positioning the mother, the patient table was slid into the mu-metal shield for data collection. The sensor array was approximately 140 cm from the mu-metal shield opening. Two 60 s runs were recorded—one with and one without the ferrite shield.The performance of the ferrite shield was assessed using four parameters: the amplitude of the fetal and maternal QRS complexes , the amplitude of the environmental noise, and the SNR. The parameters were calculated separately for each sensor and direction.To analyze the effect of the shield, the parameters must be measured from the raw signal. Thus, a 1–100 Hz band-pass filter was applied to remove the offset without removing the main interferences. The amplitude of each fQRS and mQRS was measured as the peak-to-peak amplitude. The environmental noise amplitude was defined as the root-mean-square of each interval between the end of one QRS complex and the beginning of the next. After excluding artifacts, all of the fQRS, mQRS, and noise amplitudes throughout the signal were averaged, yielding one final value for each variable. Finally, the SNR was calculated as the average fQRS amplitude divided by the average noise amplitude.We performed an ANOVA with a 2 × 2 factorial randomized block design where the patients were considered blocks (random effects). The presence of the shield and the direction, as well as the interaction between them, were considered fixed effects. All analyses were performed using the mixed procedure of SAS.Analysis and interpretation of the human data were complicated by several factors. The data show occasional inconsistencies and anomalies, such as greater-than-expected changes in the signal amplitude between the shielded and unshielded data. We largely attribute this to the fact that removing the shield requires repositioning the subject, and changes in the position and orientation of the fetus between measurements can result in inconsistencies. furthermore, for reasons we do not completely understand, maternal interference was absent in both shielded and unshielded data for two patients and was present in the shielded data but absent in the unshielded data for one patient. Statistical analysis was performed only in the five cases where maternal interference was present in both unshielded and shielded data.p < 0.001): the fQRS and mQRS amplitudes and the noise amplitude were reduced considerably, while the SNR increased significantly. The fQRS amplitude was reduced by 47% in the horizontal transverse direction and 32% in the vertical transverse direction, the mQRS amplitude was reduced by 47% and 29%, the noise was reduced by 55% and 47%, and the SNR increased by 32% and 24%, respectively. The direction did not affect the parameters, but the interaction between the presence of the shield and the direction was significant for the fQRS amplitude (p = 0.034). The multiple comparisons showed that after including the shield, the difference in the horizontal and vertical transverse fQRS amplitudes became significant (p = 0.033).The effectiveness of the ferrite shield is clearly seen in the fMCG traces shown in In the rhythm strips in In this study, we showed that placing an OPM array within a small ferrite shield can significantly improve the performance of OPM-based fMCG systems that use person-sized magnetic shields. The improvement is the result of the ferrite shield overcoming a main weakness of cylindrical mu-metal shields, namely, the low shielding factor along the longitudinal direction. As expected, the performance improvement is modest in the transverse directions but substantial in the longitudinal direction. For this reason, the shield is especially helpful when using triaxial OPM sensors, which have recently become commercially available. These sensors allow all three components of the magnetic signal to be recorded at the same location. In general, the signal components are independent, and it is necessary to measure all three to extract all the information available in the signal. This is problematic when using cylindrical, person-sized shields due to the low shielding factor in the longitudinal direction; however, as demonstrated here, the ferrite shield can compensate for this shortcoming, allowing all three channels of the triaxial output to exhibit a similarly high SNR. In a prior study utilizing dual-axis OPMs, we oriented the sensors to measure only the transverse components. This provided signals with a high SNR at the expense of the measurement of the longitudinal signal component. When using dual-axis OPMs, the ferrite shield allows flexible placement of the sensors, enabling both the transverse and longitudinal signal components to be measured. This provides a more complete characterization of the signal.In some subjects, the increase in the signal quality resulting from use of the ferrite shield provided significantly improved resolutions of the P- and T-waves. These small waveform components are crucial to rhythm assessment. The differential diagnosis of most bradycardias and supraventricular tachycardias is based on the relative timing of the atrial and ventricular activations represented by the P-wave and QRS complex. Detection of repolarization abnormalities, such as QTc prolongation in long QT syndrome or ST segment changes due to ischemia, requires resolution of the T-wave. Fetuses with congenital heart disease may also benefit from the fMCG evaluation because arrhythmia is a major cause of morbidity and mortality in this population, although arrhythmia was not present in the fetus studied here with tetralogy of Fallot. The technical improvements demonstrated in this study can provide a meaningful increase in diagnostic capability and may allow fMCG to be performed at earlier gestational ages.A particular advantage of our method is the attenuation of maternal interference, which is the dominant interference in fMCG recordings. Typically, it is not possible to shield the sensors from maternal interference and other biological interferences emanating from the subject. Instead, signal processing is relied upon. The most effective methods utilize spatial filtering, such as independent component analysis (ICA). In many cases, however, ICA cannot cleanly separate the fetal and maternal signals because their spatial characteristics, i.e., signal topographies, are not sufficiently distinct. This results in the loss of the fetal signal or the incomplete removal of maternal interference, depending on the design of the filter. The ferrite shield can mitigate this problem by attenuating maternal interference and/or altering its spatial characteristics to improve the ability of the spatial filter to distinguish it from the fetal signal.An additional advantage of the shield is improved signal visualization during data acquisition. While signal processing is highly effective in improving the signal resolution, it cannot be performed in real time. In addition to significant computation time, many signal-processing methods require user input. Visualization of the signal during data acquisition is especially important for clinical applications because it is critical to ensure that the quality and quantity of data are adequate before the patient leaves the lab.Despite our focus on fMCG, the ferrite shield is potentially useful for other biomagnetism measurements. As discussed above, it can help isolate the signal of interest from sources of biological, as well as environmental, interference. For instance, it can attenuate maternal and fetal cardiac interferences in fetal MEG and cardiac interference in magnetogastography and magnetoenterography studies. A recently demonstrated technique of simultaneously recording fMCG and fetal echocardiography —magnetomThe ferrite shield is a practical and cost-effective method of improving signal quality in comparison to the alternative methods of enhancing shielding performance mentioned in the introduction. It is simple and versatile. It can be used in conjunction with nearly any existing shield and can be easily put in place and removed as needed. Subjects report that the shield helps distribute the pressure on the abdomen and is more comfortable. Additionally, as they enter the shield feet first, their head is near the opening and they can see the outside, which minimizes the risk of claustrophobia. Using the ferrite shield may also allow the use of shorter person-sized shields, which can also help with this issue. A straightforward means of further improving the effectiveness of the method is to add a second layer of shielding. Mu-metal is much more widely used for magnetic shielding than ferrite because mu-metal is cheaper and has higher magnetic permeability. This suggests constructing a two-layer shield using ferrite for the inner shield and mu-metal for the outer shield, as implemented by Kornack and coworkers .In conclusion, we demonstrated the use of a small ferrite shield as a simple, practical method of overcoming the high level of environmental interference present in fMCG recordings made using person-sized magnetic shields. The method allows users to take full advantage of newly available OPM sensors that record all three components of the signal and provide meaningful improvements in signal resolution and diagnostic capability. This can help make fMCG more widely available as a new technique for evaluating life-threatening fetal arrhythmia and other high-risk pregnancy conditions associated with abnormal fetal heart rate and rhythm.


In the present study, various combinations of dimensionality reduction methods with data clustering methods for the analysis of biopsy samples of intracranial tumors were investigated. Fresh biopsies of intracranial tumors were studied in the Laboratory of Neurosurgical Anatomy and Preservation of Biological Materials of N.N. Burdenko Neurosurgery Medical Center no later than 4 h after surgery. The spectra of Protoporphyrin IX (Pp IX) fluorescence, diffuse reflectance (DR) and Raman scattering (RS) of biopsy samples were recorded. Diffuse reflectance studies were carried out using a white light source in the visible region. Raman scattering spectra were obtained using a 785 nm laser. Patients diagnosed with meningioma, glioblastoma, oligodendroglioma, and astrocytoma were studied. We used the cluster analysis method to detect natural clusters in the data sample presented in the feature space formed based on the spectrum analysis. For data analysis, four clustering algorithms with eight dimensionality reduction algorithms were considered. Recommendations for neuro-oncologic surgery include maximal tumor resection with minimal risk of functional complications with mandatory use of preoperative planning, including functional magnetic resonance imaging and tractography, microsurgical techniques, and intraoperative optics . ReliablCurrently, a multimodal strategy in the surgery of malignant brain tumors using electrical cortical stimulation, neuronavigation, and fluorescence is generally accepted ,5,6,7. TThe most widely used method for intraoperative demarcation of tumors in neurosurgery today is the video analysis of Protoporphyrin IX (Pp IX) fluorescence in a special excitation mode with the violet light of an operating microscope. However, this method is only suitable for tumors with a high accumulation of fluorescent tumor markers (such as Pp IX). If the marker does not accumulate in the area of interest, which occurs in 40% of benign tumor surgeries , the surRecently, several research groups have also focused their interest on the combination of different optical–spectral methods. For example, in the most recent study, a combination of histology based on stimulated Raman scattering and analysis of an autofluorescence signal with two-photon excitation was considered . The autThe success of automatic classification methods in this application of optical spectroscopy leaves open the question of how exactly the optical–spectral signatures of tissues correspond to their morphological characteristics since all automatic classification methods are supervised methods; that is, they work on the basis of preliminary labeling of data based on the results of pathomorphological examination. Our study is devoted to the preliminary analysis of spectral data using clustering methods. These are methods for grouping unlabeled data in a given feature space on the basis of their similarity/proximity, according to some metric of the distance between them. The most commonly used metric is the Euclidean distance, but other metrics such as the Manhattan distance or the Minkowski distance can also be used. This procedure is interesting because it non-subjectively groups input observations  based on the natural similarity of their properties . Therefore, in our study, we aimed to find naturally occurring clusters of some brain tumor tissues in the spectral feature space.The authors of  emphasizThe authors of  showed tIn this study, we propose to analyze the crucial differences in the appearance of the data in different feature spaces after dimensionality reduction by different techniques . In FiguIf we consider which spectral features have the largest representation in the new axes after linear discriminant analysis b, we seeFor nonlinear methods of dimensionality reduction , the visWe also consider various options for combining dimensionality reduction techniques with cluster analysis. For each method of clustering, we have chosen the two best approaches of dimensionality reduction, taking into account (a) the clustering quality metrics (Silhouette and Davies–Bouldin indices) and (b) the distribution of samples with different diagnoses into separate clusters, so that there is at least one clear cluster with preferably one diagnosis.The best k-means clustering in terms of combining the maximum of the Silhouette Index (SI) and the minimum of the Davies–Bouldin Index (DBI)  was obtained for the locally linear embedding dimensionality reduction technique a,b. HoweFor LDA with k-means, we can see c,d that Looking at the spectral characteristics inherent to the clusters on average, for the combination of k-means and LDA, we see the high level of carotenoids and oxyhemoglobin as the main differences for the 3rd cluster (represented mainly by meningiomas). Moreover, the first two clusters, populated by gliomas with varying degrees of malignancy, show differences both at the level of elastic light scattering (which is much higher in the first cluster) and in the content of lipids (higher in the first cluster) and hemoglobin (lower in the first cluster). Thus, based on its molecular characteristics, the first cluster is more likely to contain more benign glial tumors and areas from the perifocal zone of glioblastoma. To further investigate these differences, it is planned to conduct a more thorough analysis of the pathological picture of the corresponding samples.From the point of view of clustering metrics, the best result was achieved when applying hierarchical clustering with LLE with the number of clusters equal to 3 a,b. HoweOn the other hand, agglomerative clustering with linear methods of dimensionality reduction yielded interesting results on the distribution of pathological diagnoses between clusters c,d. If wIn this study, for the second cluster (meningiomas) obtained with clustering after LDA, we see a distribution of molecular features similar to For the DBSCAN method, the SI was maximum for the LLE and HLLE methods, but the DBI was also quite high for them. The SE method showed the best combination of indices.In However, this method failed to identify a cluster predominantly populated by meningiomas, forcing us to conclude that it is not suitable for our data distribution.Since in fuzzy clustering, we do not have an unambiguous correspondence between an object and a cluster but only probabilities of belonging, we assume that an object is in a cluster if its probability of being in it is maximal compared to its neighbors. It is also important to consider the probability of finding each type of tumor in each cluster. To calculate the probability of occurrence of each diagnosis in the cluster, the probabilities of occurrence of each object in that cluster were summed up, normalized to 100%, and then multiplied by the resulting conversion factor.ij the probability of an object i belonging to the cluster j. Then, for each cluster, the conversion factor is calculated as follows:Let us consider Pdj, summing the shares of each object in the cluster for each diagnosis:And then we recalculate the probability of belonging of each object to shares (fraction) for each diagnosis (diagnosis coefficient d) FIn terms of clustering objects with different diagnoses, the linear discriminant analysis method performed best. The third cluster in this partitioning is a good candidate to find meningioma . The senAmong the clustering methods considered, methods such as k-means, hierarchical clustering and c-means provided a fairly uniform distribution of objects into clusters. The DBSCAN method considered our data as practically a single cluster, from which we can conclude that it is not applicable to our distribution of features.The use of different dimensionality reduction methods has shown different results in terms of clustering characteristics such as silhouette index, etc., and in terms of the distribution of different diagnoses into different clusters . In the Since cluster analysis works on unlabeled data, the separation of glial and meningeal tumors into separate clusters indicates their natural differences in the considered feature space formed as a result of the identification of characteristic bands in Raman, fluorescence, and diffuse reflectance spectra. The obtained results can be used both for the recovery of missing data and for the stage of data preprocessing when using classification algorithms on labeled data.Our results are in good agreement with the results of , despiteA He-Ne laser  was used to excite a fluorescence signal from biopsy samples, which was observed using LESA-01-BIOSPEC fiber-optic spectrometers  with optical edge filters to attenuate the excitation radiation (632.8 nm) at the spectrometer input to the level of the fluorescence signal. A halogen lamp was used as a white light source to obtain a diffuse reflectance signal. The spectra were recorded using Uno software version 1.0.7  installed on a laptop computer. The exposure time was 100 ms for the fluorescence signal, and 10 ms for the diffuse reflectance signal.−1 resolution. The laser radiation and the Raman scattered light were delivered using a confocal fiber-optic probe. The laser power was 150 mW measured at the output of the laser source. The Raman spectra of each sample were recorded in a series of 10 measurements with 30 s exposure for each measurement. The background signal was measured prior to each series. Both the background and the Raman spectra were measured in a darkened room [Raman scattering was excited with a 785 nm Ramulaser-785 laser  and observed with a Raman-HR-TEC-785 fiber-optic Raman spectrometer  with 4 cmned room .The studies were conducted in the Laboratory of Neurosurgical Anatomy and Preservation of Biological Materials of N.N. Burdenko Neurosurgery Medical Center  on tumor tissue samples removed during neurosurgery and placed in saline immediately after removal. The minimum sample size was 3–4 mm. After the spectral measurements, the samples in formalin were sent for pathomorphologic examination. Samples from 54 patients with diagnoses of glioblastoma (n = 28), meningioma (n = 12), astrocytoma (n = 9), and oligodendroglioma (n = 5) were examined. Between 1 and 4 biopsy specimens  were taken from each patient, followed by verification by pathomorphologic examination. Raman spectra, fluorescence of 5-ALA-induced protoporphyrin IX, and diffuse reflectance spectra in the 500–600 nm region were measured for all samples.Registration of diffuse reflectance spectra in white light and fluorescence spectra with 632.8 nm laser as an excitation source using the LESA-01-BIOSPEC spectrometer;Registration of the spontaneous Raman spectra of a sample excited at 785 nm using the Raman-HR-TEC-785 spectrometer.Optical–spectral measurements of each biological sample were performed in two steps including :RegistraFeature filtering removes features  that may contain noise or non-valuable information. This procedure allows us to reduce the dimensionality of the data and focus on the valuable information. The authors of  found thDimensionality reduction techniques can be divided into linear and non-linear. Linear methods include principal component analysis (PCA) and linear discriminant analysis (LDA). Non-linear methods include multidimensional scaling (MDS), isometric mapping of objects (Isomap), local linear embedding (LLE), Hessian locally linear embedding (HLLE), spectral embedding , and t-distributed stochastic neighbor embedding (t-SNE).PCA is used to project a higher-dimensional data matrix onto a low-component subspace. It reduces the set of variables to a smaller set of orthogonal, and thus independent, principal components in the direction of maximum variation, i.e., it reduces the dimensionality and preserves the most significant information for further analysis.The main goal of linear discriminant analysis  is to find “axes of discrimination” that optimally classify data into two or more classes. LDA is closely related to PCA  in that both look for latent axes that compactly explain the variance in the data.The main difference between PCA and LDA is that LDA is a supervised method and PCA is an unsupervised method. PCA looks for predictions that maximize the variance, and LDA looks for predictions that maximize the ratio of interclass variance to intraclass variance . The datMultidimensional scaling (MDS) seeks a low-dimensional representation of the data in which the distances closely match the distances in the original multidimensional space.Isometric feature mapping (Isomap) projects the data into a lower dimension while maintaining geodesic distance (rather than Euclidean distance as in MDS). The geodesic distance is the shortest distance between two points on a curve.Locally linear embedding (LLE) looks for a lower-dimensional projection of the data that preserves distances within local neighborhoods. It can be thought of as a series of local principal component analyses that are compared globally to find the best non-linear solution.Hessian local linear embedding projects the data into a lower dimension, preserving local neighborhoods like LLE, but uses the Hessian operator to better achieve this result.Spectral embedding uses spectral methods to reduce dimensionality by matching adjacent inputs and adjacent outputs. This preserves locality, but not local linearity.t-SNE converts the affinities of data points into probabilities. Similarities in the original space are represented by Gaussian joint probabilities, and affinities in the nested space are represented by Student’s t-distributions. This allows t-SNE to be particularly sensitive to local structure.Clustering is the process of grouping a set of physical or abstract objects into classes of similar objects. We can say that clustering is successful if the data in one cluster is very similar in one (or more) parameter, and has little similarity between data from other clusters.hard clustering: each object belongs to only one of the clusters;fuzzy clustering: each object belongs to each cluster to some extent; i.e., the algorithms fix the probabilities of a given object belonging to each cluster, and these probabilities must sum to “1”.Cluster analysis methods can be divided into two main categories:centroid-based clustering;hierarchical clustering;density-based clustering;distribution-based clustering.Also, we can consider another approach to classify them based on the mechanism of grouping data:The k-means algorithm is the simplest and one of the most widely used cluster analysis methods. In addition to the data, this method requires additional input parameters, such as the number of clusters (K) and the initial center of the cluster.In this work, we also used agglomerative hierarchical clustering. There are two main types of hierarchical clustering algorithms: agglomerative (bottom-up) and divisive (top-down) approaches. Divisive algorithms work according to the top-down principle: at the beginning, all objects are placed in one cluster, which is then divided into smaller and smaller clusters. More common are bottom-up algorithms, which start by placing each object in a separate cluster, and then the clusters are merged into increasingly larger clusters until all sample objects are contained in a single cluster.DBSCAN  is a widely used algorithm for distinguishing closely spaced but fairly dense clusters of complex shapes.When using fuzzy clustering methods, each observation eventually belongs to more than one cluster with a given probability. One of the widely used soft clustering algorithms is the fuzzy c-means clustering (FCM) algorithm, which gives better results for overlapping data sets compared to k-means clustering.For hard clustering algorithms, we used three metrics of clustering quality: Silhouette Index (SI), Calinski–Harabasz Index (CH), and Davies–Bouldin Index (DBI).The Silhouette Index is the average silhouette coefficient for all clusters, calculated using the average distance within a cluster and the average distance to the nearest cluster. This score ranges from −1 to 1, where the higher the score, the more well-defined and distinct the clusters are.The Calinski–Harabasz Index is calculated using inter-cluster variance and intra-cluster variance to measure differences between groups. As with the Silhouette Index, the higher the score, the more clearly defined the clusters are. This score is boundaryless, meaning that there is no “acceptable” or “good” value. However, it can be used to compare different clustering methods.The Davies–Bouldin Index is the average similarity of each cluster to its most similar cluster. Unlike the previous two metrics, this score measures the similarity of clusters, meaning that the lower the score, the better the separation between clusters.For soft clustering evaluation, on the other hand, we used its own metrics based on the probability distribution of data in clusters: Partition Coefficient (PC) and Partition Entropy Coefficient (PEC). The PC measures the degree of homogeneity within each cluster, whereas the PEC measures the degree of heterogeneity between clusters. It is defined as the negative sum of the product of the fraction of data points in each cluster and the logarithm of the fraction of data points in each cluster.Both PC and PEC range from 0 to 1, with higher values indicating better clustering results. A PC value close to 1 indicates that the data points are well clustered, whereas a PEC value close to 0 indicates that the clusters are very heterogeneous.Since we observe a noticeable separation in the space of reduced dimensionality of objects defined by the pathomorphologist as meningiomas against the different glial tumors, it is reasonable to introduce parameters demonstrating how well meningiomas are distinguished into a separate cluster using different methods. For this purpose, in each partition, we call the clusters in which meningiomas predominate over glial tumors meningioma clusters. And in this case, we calculate the sensitivity of the clustering method for meningiomas  and the specificity .Obviously, there is no need to make this analysis meningioma-centric; we can consider the results in terms of the identifiability of glial tumors. Since they are fairly evenly distributed in clusters, it makes sense to consider them in general. The sensitivity of meningioma detection corresponds to the specificity of glioma detection, and vice versa.In this study, cluster analysis methods were applied for statistical processing of data obtained from biopsy specimens of intracranial tumors. The data were obtained using Raman, fluorescence and diffuse reflectance spectroscopy. An optimal combination of dimensionality reduction and cluster analysis techniques was sought to detect natural differences between samples in the spectral feature space. Both linear (PCA and LDA) and non-linear approaches  were investigated. Cluster analysis was also performed using several methods, including hard clustering methods such as k-means, agglomerative hierarchical clustering and DBSCAN, and fuzzy c-means clustering technique.After considering all possible combinations of the mentioned methods, we selected the two best dimensionality reduction algorithms for each clustering algorithm by considering two approaches to assessing the quality of clustering. The first approach was based on the clustering quality metrics (Silhouette and Davies–Bouldin Indices) and the second approach used the information about the distribution of samples with different diagnoses into separate clusters so that there would be at least one clear cluster with preferably one diagnosis.From the point of view of the first approach, the best results were obtained using the agglomerative hierarchical clustering algorithm with the local linear nesting algorithm as the dimensionality reduction method. However, the sensitivity of this clustering to different diagnoses was low. The second approach to assessing the clustering quality gives us the optimal combination of LDA with k-means  and the second-best combination of LDA with agglomerative clustering .The results of the study of the clustering algorithms of the obtained data allow us to conclude that they can be naturally divided into subsets of meningeal and glial tumors. For further differentiation of glial tumors, we plan to continue the study using classification methods on labeled data.