Osteoarthritis (OA) is a risk factor for falls. To decrease the fall risk, it is important to evaluate the detailed features of the gait of patients with OA. This study aimed to investigate the spatio-temporal parameters of gait in patients with end-stage hip OA, especially foot trajectory. We measured normal speed gait in patients with hip OA and in healthy controls (HCs) using inertial measurement units attached to shanks. The stride lengths in the affected and unaffected sides in the patients with hip OA were shorter than those in the HCs, but the position of maximum foot clearance was not significantly different between the two groups. The patients with hip OA compensated the position of maximum foot clearance to avoid fall risk. The horizontal plane foot trajectory in patients with hip OA suggests that the lateral bending of the trunk during walking, which is frequently seen in them, was a result of the lateral distance at swing down being located significantly more medially on the unaffected side than on the affected side. Herein, a new gait parameter of lateral distance at swing was discovered by a detailed evaluation of three-dimensional foot trajectory of female patients with end-stage hip OA. Limited range of motion (ROM), inguinal pain and limping are characteristic5, and their effects cause activity limitation6. The patients with hip OA have a higher risk of death than healthy individuals, and the history of diabetes, cancer, or cardiovascular disease and gait disorders are listed as major risk factors7. In addition, OA is also known to be a risk factor for falls8. However, it is not well understood what kind of walking problem the OA patients have. The association between fall risk and pain and dysfunction has been modest and not associated with radiographic OA9. In addition, female patients with end-stage hip OA have a higher incidence of falls within 1 year, and an association between fall risk and limping and knee extensor strength has been reported, but detailed gait evaluation has not been performed10. To decrease the fall risk during walking, it is important to understand how hip OA affects gait features in detail. In addition, recently, it has been reported that abnormal hip joint loading is involved in the progression of hip OA11. The hip joint loading is potentially modifiable with interventions using gait training13, which requires appropriate gait instruction and adjustment of gait volume. For the appropriate instruction, it is also important to evaluate the gait of the patients with hip OA accurately and in detail.Hip osteoarthritis(OA) is common in females17. When walking on the treadmill at constant speed, the step length on the affected side is longer than that on the unaffected side.20. Many researchers reports on the walking speed of patients with hip OA22; however, reports comparing the stride speed of the affected and unaffected limbs are rare. There are various phases in the gait cycle such as swing and stance; the gait is generated by the rhythmic combination of these phases. Patients with hip OA have an asymmetry in step and swing duration between affected and unaffected limbs caused by short step22 and stance durations23 in the affected limb. The increase in step length asymmetry is associated with a decrease in mechanical energy exchange20. In addition, the external hip flexion, extension, and adduction moments during walking are lower in patietns with hip OA24, and extension angles of the hip and knee in the latter half of the stance phase are smaller than those in healthy controls(HCs), but there is no difference in the ankle joint angle25. The peaks of hip flexion and extension angles decrease in the affected limb relative to those in the unaffected limb26. In patients with hip OA, the variability of walking on the treadmill was higher on the affected side than in the HCs, but the variability was lowest when the gait speed was close to the self-selected walking speed27. The variability of stride duration decreased significantly 1 year after total hip arthroplasty (THA) compared with that preoperatively and was comparable to that of the HCs28.Regarding gait characteristics of patients with hip OA, it is well-known that walking speed decreases due to short stride length and short affected limb stepAs seen in these previous studies, various gait characteristics of hip OA have been reported. However, to the best of our knowledge, there is no report that investigates the gait characteristics of patients with hip OA in detail using three-dimensional foot trajectory. Since the gait parameters of patients with hip OA are asymmetric the foot trajectory is likely also asymmetric. The three-dimensional foot trajectory shows features in the sagittal, frontal, and horizontal planes, but gait features in the horizontal plane were not almost reported. Patients with hip OA frequently show an exaggerated lateral bending of the trunk during gait. The lateral bending of the trunk may be characteristic of the foot trajectory, especially during the swing phase.1. Male patients were excluded in this study to avoid possible confounding due to unbalanced sex distribution. The evaluation of three-dimensional foot trajectory provides multiple gait features such as the sagittal, horizontal, and frontal planes of the spatio-temporal parameters. In recent years, there has been an increase in gait analysis with wearable inertial measurement unit (IMU) sensors that have provided an accessible and affordable alternative to traditional optical gait analysis systems29. Mao et al. proposed a system to estimate the foot trajectory in three dimensions during walking by attaching the IMUs on shanks just above both ankles30. The gait indices such as the stride length, the position of maximum foot clearance (maximum foot clearance), and speed estimated by this system strongly correlate with those measured by the optical motion-capture system. This system can also evaluate stride duration, stance duration and swing duration for the right and left lower limbs. In this study, we use this wearable system to evaluate the gait characteristics of patients with hip OA.This study aimed to evaluate the three-dimensional foot trajectory of female patients with end-stage hip OA on the affected and unaffected limbs and to clarify the gait characteristics in comparison with HC. The average prevalence of hip OA in Japan is 1.0%, but is higher in females (2.0%) than males (0%)The participants were 50 outpatients with unilateral end-stage hip OA at the Department of Orthopedic Surgery, Nissan Koseikai Tamagawa Hospital,  in 2018-2019. We calculated that with a sample of 75 participants, the study would have 80% power to detect a 0.17 effect size, with a type I error rate of 5%. The patients had a minimal joint space (MJS) of less than 2 mm and a low Japan Orthopaedic Association (JOA) hip score, and were considered severe enough to require THA. We defined these patients as end-stage. The HC consisted of 25 age-matched females who were independent in their daily activities and able to walk on their own. We excluded the patients if they had neurological, vascular, and other lower extremity musculoskeletal conditions or psychiatric disorders that affected gait or functional performance. We also excluded patients with self-reported lack of sensation in the foot or lower limb and those who were unable to walk short distances (30 m) without an assistive device and patients with pain in the hip on the unaffected side. This study was approved by the Institutional Review Board of Nissan Tamagawa Hospital and the Tokyo Institute of Technology for research ethics committee. All participants provided written informed consent before participation. All methods were performed in accordance with the Helsinki Declaration and relevant guidelines and regulations. There were no differences in age, height, weight, or BMI between patients with hip OA and HCs Table . There w31 when passive ROM was measured by an experienced physical therapist. The measuring was conducted with the patients in the supine position except for measuring the external and internal rotation angles. The external and internal rotation angles were measured in the prone position at 9032. Hip pain during walking was measured using the visual analogue scale. The stage of hip OA was classified by orthopaedic surgeons based on the Japanese Orthopedic Association’s osteoarthritis staging system.The hip joint ROM angles were determined by passively moving the patients’ legs and measuring the maximum angles using a goniometer. The pelvis was stabilised to prevent rotation or tiltingThe IMUs  that were used for measured gait are shown in Fig. 30. Spatio-temporal parameters were estimated via the double integration of the linear acceleration transformed by the IMU orientation information. To reduce the integral drift error, an inverted pendulum model, applied with a linear error model, was introduced at the mid-stance to estimate the update velocity30. The motion of the IMU at the mid-stance can be modelled as a rotational motion in a three-dimensional space. The position vector r and angular velocity in the laboratory coordinate frame were computed from the estimated orientation of IMU30. By this method, we estimated the maximum foot clearance, stride length, stride speed, stride duration, stance duration and swing duration for left and right lower limbs.We estimated spatio-temporal gait parameters using the method proposed by Mao et alFor the measurements, participants walked about 30 m down a straight and flat corridor at their self-selected speed. The measurement was carried out with participants familiar shoes with flat soles. The number of left and right strides were measured; at least 30 for each stride. All measurements were taken once for each participant. All participants walked without any assistive devices.pWe used the fourth to the 13th stride of each leg to calculate means for each gait parameter. We aggregated each parameter for the left and right legs instead of affected and unaffected sides to simply compare the results of the LOA and ROA patients with the HCs. The mean of each spatio-temporal parameter and lateral distance at swing phase were analysed using two-way mixed ANOVA. The factors for the ANOVA were three groups  and two legs (left and right legs). For multiple comparisons, we used Shaffer’s method. We used the Student’s T-test and the Mann-Whitney U test for the participant demographics, affected/unaffected physical functions and JOA hip score with and without normal distribution, respectively. In addition, the stage of hip OA was analysed using the chi-square test. All statistical analyses were performed using R with the significance level set at An example of each group in the three-dimensional foot trajectory of the left leg during walking is shown in Fig. Due to space limitation, we show only important statistical results. All results of the statistical test are shown in the supplementary information.post hoc test for these gait parameters. When the simple effect of the group was found, we conducted a multiple comparison using Shaffer’s method. The results are shown in Table The spatial gait parameters of the left and right lower limbs for HC, LOA, and ROA, i.e., maximum foot clearance and stride length are shown in Fig. entclass1pt{minimapost hoc test for these LDs of each swing phase. The results are shown in Table For the LD at swing phase of spatial gait parameters, the mean, maximum and minimum LDs of the left and right lower limbs of the HC, and the patients with LOA and ROA from the horizontal plane during toe off, maximum foot clearance, and swing down are shown in Fig. post hoc test for these gait parameters. When the simple effect of the group was found, we conducted a multiple comparison using Shaffer’s method. The results are shown in Table The temporal gait parameters of the left and right lower limbs for the HC, patients with LOA and ROA, speed, stride duration, stance duration, and swing duration, are shown in Fig. entclass1pt{minimaentclass1pt{minimaThis study investigated in detail the spatio-temporal parameters in three-dimensional foot trajectory features of female patients with end-stage hip OA using the IMUs attached to the shanks on the affected and unaffected limbs. The results were summarised and the speed of each stride, maximum foot clearance, and the minimum LDs of the foot during swing phase were especially highlighted either for their differences from previous studies or for their novel perspectives.In the summary of spatial gait parameters, there was no difference in the maximum foot clearance between the patients with hip OA and the HCs while the stride length was significantly shorter in the patients with OA. In addition, in patients with hip OA, the stride length was significantly shorter in the affected side compared to those in the unaffected side. There was no difference in the mean, maximum and minimum LDs of the foot during the swing phase between patients with hip OA and HCs in toe off and maximum foot clearance and swing down. However, the unaffected sides were located more medially than the affected sides at the swing down minimum.For the temporal gait parameters, the speed of patients with hip OA was significantly slower, and their stride duration, stance duration, and swing duration were significantly longer in the patients with hip OA than those in the HCs. In addition, in patients with hip OA, the speed, and stance duration were significantly decreased on the affected side compared to those on the unaffected side; and the swing duration was significantly longer on affected side than that on the unaffected side. However, there was no significant difference in stride duration between the affected and unaffected sides.33. On the other hand, the results for the speed of each stride and the maximum foot clearance differed form those in some previous studies. In measuring the speed of each stride in the left and right lower limbs, there was a significant difference in the stride speed between the affected and unaffected side in the patients with LOA and ROA. This lower speed in the affected side would be caused by a decrease in hip moment24 and pain, resulting in a decrease in affected stride length. However, a previous study showed no significant difference in speed between the affected and unaffected sides34. This difference between that study and ours is likely caused by the higher radiographic severity of the patients in our study than that in the previous one. Thus, the slower stride speed in the affected side compared with the unaffected side would be associated with heavy radiographic severity. In fact, the hip moment loading asymmetry is associated with OA radiographic severity. Foucher et al. demonstrated that with increasing Kellgren-Lawrence grade, subjects exhibited a greater degree of gait abnormalities known as motion discontinuity35.Some of the results basically supported previous studies. The results for stride length, speed, stance duration, and swing duration were consistent with gait characteristics of individuals with hip OA in the previous study36. From this perspective, we might expect a decrease in the maximum foot clearance. However, the results of this study revealed no significant difference in the maximum foot clearance between the patients with hip OA and HCs, though their stride lengths were shorter than the HCs. Thus, regardless of the small hip extension-flexion angle, the patients with hip OA could raise their foot as high as HCs. In the future, the hip extension-flexion angle and foot trajectory of patients with hip OA during gait should be measured at the same time.In the patients with hip OA, the hip extension-flexion angle during gait decreases17. This large pelvic tilt could guarantee the maximum foot clearance as high as in healthy individuals. In addition, knee joint angle during walking increases in patients with hip OA compared with the HC25, which could also guarantee the maximum foot clearance. Another possibility for maintaining the maximum foot clearance could be increased strength of foot muscles. Although the degree of the ankle joint movement was not different in the patients with hip OA compared with that in the HCs25, the ankle dorsiflexion increased in both the affected and unaffected sides of the patients with hip OA compared with the HCs. In addition, the electromyogram of the tibialis anterior muscle in the unaffected side also increased in the patients with hip OA37. These possibilities should be investigated by measuring more indices relating to gait, including the foot trajectory.One possible explanation for patients keeping the max foot clearance is as follows. The sagittal view of the pelvis during push-off of hip OA showed 2.5 times more pelvic upward tilt than that of the nonclinical subjects38. However, this compensation causes the centre of gravity to move to the affected side. Thus, this shift of the centre of gravity is one possible reason for medial LD at swing down minimum in patients with hip OA.Moreover, the trajectory analysis for the lateral direction importantly suggested a new perspective. LD at the swing down minimum in patients with hip OA was located significantly more medially on the unaffected side than on the affected one. In the frontal plane, there was no significant difference between affected and unaffected sides with respect to hip adduction angle. Thus, the medial LD at swing down minimum would not relate to the ROM of the angle. The medial LD could relate to an exaggerated lateral bending of the pelvic and trunk to affected side during gait in hip OA patients, which is called a Duchenne limp. The Duchenne limp is a movement performed in order to compensate for weakened hip abductor muscles in the affected side to maintain a stance on the affected leg39 and peak hip extension angle of the affected limb decrease26. Thus, the hip external rotation  movement could also decrease the anterior coverage of the femoral head during terminal stance, and could be compensated by pelvic rotation, which may lead to a medial trajectory of the swing phase.However, the present experiment specifically showed that there was no significant difference in the LD at mean and maximum, but LD at swing down minimum for patients with hip OA was located significantly more medially in the unaffected limb than that in the affected limb. When the unaffected limb swings down in the swing phase, the affected limb is at the terminal stance phase. In patients with hip OA, the anterior femoral head coverage10. The causes of tripping have been discussed from the viewpoints of the foot trajectory. For example, the minimum foot clearance (MFC) and minimum toe clearance (MTC) have been reported as risk factors for falls41. On the contrary, the causes of loss of balance have not been considered within the context of the foot trajectory. The gait features in LD are potentially related to the loss of balance due to the increased sway of the centre of gravity to the left and right. It could be important to instruct the patient to walk with the swing of the unaffected side limb slightly outward to avoid falling. In the future, it is necessary to investigate whether this gait instruction is effective in preventing falls in patients with hip OA and whether this affects the function of the affected side limb.The incidence of falls within 1 year is high in female patients with end-stage hip OA, and the main causes of falls are tripping (43.5%) and loss of balance (37.0%)This study aimed to investigate the spatio-temporal features, especially, the foot trajectory, in gait parameters of patients with end-stage hip OA using IMUs attached on both shanks. The stride length of the patients with hip OA was significantly shorter on both the affected and unaffected sides than that of the HCs; but maximum foot clearance was not significantly different between the patients with hip OA and HCs. Thus, the foot trajectory of the patient with hip OA was less than the HC only in the forward direction. This result suggests that hip OA compensates the maximum foot clearance in gait as much as healthy people to decrease fall risk. The horizontal and frontal planes foot trajectories are almost symmetrical in HCs, but appear asymmetrical in patients with hip OA. In the horizontal plane foot trajectory of patients with hip OA, the mean LD from the midline of toe off, maximum foot clearance, and swing down during the swing phase were not significantly different from those of HCs, but the swing down minimum of the swing phase in patients with hip OA was located significantly more medially on the unaffected side than on the affected one. The stride length and speed were significantly decreased in both affected and unaffected sides compared with the HC. The stride length and speed were significantly lower in the affected side than in the unaffected side. The stride duration was significantly longer in the hip OA than those of the HC. The stance and swing durations were shorter and longer in the affected side than in the unaffected side. These temporal gait parameters were consistent with gait characteristics of individuals with hip OA in a previous study. In this study, a new gait feature was discovered by detailed evaluation of three-dimensional foot trajectory of female patients end-stage hip OA. In the future, it is necessary to clarify the relationship between these gait characteristics, clinical symptoms, and physical functions.Supplementary Information.


In recent years, the use of deep learning-based models for developing advanced healthcare systems has been growing due to the results they can achieve. However, the majority of the proposed deep learning-models largely use convolutional and pooling operations, causing a loss in valuable data and focusing on local information. In this paper, we propose a deep learning-based approach that uses global and local features which are of importance in the medical image segmentation process. In order to train the architecture, we used extracted three-dimensional (3D) blocks from the full magnetic resonance image resolution, which were sent through a set of successive convolutional neural network (CNN) layers free of pooling operations to extract local information. Later, we sent the resulting feature maps to successive layers of self-attention modules to obtain the global context, whose output was later dispatched to the decoder pipeline composed mostly of upsampling layers. The model was trained using the Mindboggle-101 dataset. The experimental results showed that the self-attention modules allow segmentation with a higher Mean Dice Score of 0.90 ± 0.036 compared with other UNet-based approaches. The average segmentation time was approximately 0.038 s per brain structure. The proposed model allows tackling the brain structure segmentation task properly. Exploiting the global context that the self-attention modules incorporate allows for more precise and faster segmentation. We segmented 37 brain structures and, to the best of our knowledge, it is the largest number of structures under a 3D approach using attention mechanisms. The scientific community has developed tools that allow for obtaining brain information so doctors can study the human brain. Among these tools we find brain imaging, which includes methods such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and ultrasound (US), among others. However, not all methods produce quality images when applied to the brain because high contrast images are required to study the human brain. Therefore, highly sensitive methods such as magnetic resonance imaging or positron emission tomography must be used .Although PET scans are capable of obtaining good quality images, they are not the preferred choice for specialists as they have several disadvantages. PET scans cannot reveal structural information at the microscopic and macroscopic levels in the white and gray matter of the brain; cannot detect changes in brain activation, and pose health risks due to the required radiation . For theTraditionally, interpretations of medical images have been made by human experts. However, the existence of variations in the criteria among various human specialists is a limitation in relation to the generation of an efficient and precise diagnosis . This weIn general, multiple methods for the segmentation of brain magnetic resonance images have been proposed. These methods can be grouped as follows: manual methods, spatial dimensionality-based methods (2D and 3D), pixel/voxel intensity-based methods, atlas-based methods, surface-based methods, and methods based on deep learning techniques .The manual segmentation of magnetic resonance images is based on the use of highly trained personnel to obtain the different types of brain tissues. In order to perform segmentation, experts commonly use manual delineation tools that allow them to delineate different regions of the brain. Some examples of these tools are FreeSurfer , BrainSuIn terms of the spatial dimensionality methods, these are subdivided into 3D and 2D approaches. Three-dimensional-based segmentation approaches seem to be the natural way to approach the problem because it allows to exploit the three-dimensional nature of MRI by considering each voxel and its relationship to neighbors at different acquisition planes . However, the 3D approach still has limitations, mainly related to the high computational cost in computers with limited memory, the increase in the complexity of the models and the number of parameters, making the learning process slower ,21,22. TIntensity-based methods attempt to find a threshold value that separates the different tissue categories. These methods include techniques of thresholding, growth of regions, classification, and grouping . In 23,,23,24, aWorks that use region growth techniques, use the similar characteristics of pixels found together to perform the separation of a common region . Region Atlas-based methods are those that make use of brain anatomical information from a specific population for image segmentation. Likewise, in ,32,33,34Several works have also been proposed that make use of deformable models. These techniques are part of the group of surface-based methods where the main objective is the delimitation of regions with similar characteristics through the use of elastic curves . SimilarRecently, deep learning has become an area of interest in the scientific community due to the important results that have been achieved in multiple disciplines ,41,42,43Convolutional neural networks are a type of neural network that was created by LeCun but was inspired by Fukushima’s work on the neocognitron for the recognition of handwritten Japanese characters . In the Similarly, the authors of  were ablSome of the most important deep learning solutions have been proposed using a 2D representation allowing researchers to segment more structures than a 3D representation allows them to do. In fact, the segmentation of more than 25 brain structures into a 3D representation has been achieved by a few works, while using a 2D representation, deep learning works can segment more than 95 brain structures .The use of this type of neural network is not limited to the segmentation of brain tissues or brain structures. These have also been used in the segmentation of brain lesions, as in ,56,57, tOne of the architectures most commonly used in medical image segmentation tasks is the U-Net architecture . Due to Multiple neural network architectures based on U-Net have been proposed for the field of medical image segmentation. The primary goals of these works were to improve the network by using skip connections between the layers of the coding and expansion path ,65 and tHowever, convolutional neural networks have serious limitations. One of them is the loss of image characteristics due to pooling operations . This isOn the other hand, researchers have proposed multiple deep learning architectures based on attention mechanisms such as the Transformer’s architecture ,23,68. TThus, the scientific community has achieved that the Transformer’s architecture can obtain results comparable to those established as the state of the art in computer vision methods . In factAlso, another Transformer-based architecture is the Medical-Transformer network , which iIn this study, a 3D architecture of deep neural networks is proposed for the task of segmenting volumes associated with brain structures from MRI. Our proposal uses an encoder/decoder approach, strengthening the connection between them by incorporating self-attention modules and skip connections. The attention modules as well as the convolution operations allow the network to incorporate global and local characteristics, and achieve a more detailed segmentation of the edges of structures.We used the publicly available Mindboggle-101 dataset . This daWe performed some preprocessing steps. We created an array representation of the original MRIs and segmentation masks in the dataset making it straightforward for training deep learning models. Additionally, for the selected structures, we created a label remapping strategy with IDs 1 to 37 having the ID 0 set for background. We created the segmentation ground truth files by taking the aseg+aparc files and mapping the existing IDs to our desired IDs. After this, we applied a min-max scaling to the voxel values to be in a range between 0 and 1. This intensity rescaling was performed using the histogram equalization technique over each individual MRI.Then, we applied a filter in each MRI where empty slides were removed from the brain volumes, leaving on average 192 slides per brain plane. These preprocessed MRI volumes were divided into nonoverlapping subvolumes of size 64 × 64 × 64 voxels that were saved in single files containing stacks of volumes per brain MRI see . FinallyThe computational implementations were performed with the open source library for numerical computation Tensorflow and run on a computer with a 5th generation Intel I7 4820k@3.70 GHz processor, 64 GB of RAM memory, and two Nvidia 1080TI video cards with 11 GB of GDDR 5x RAM at 405 MHz.The proposed deep neural network architecture is structured as an encoder-decoder architecture. The contracting path follows the typical architecture of a convolutional neural network. However, we applied Transformer layers at the end of this path using the extracted feature maps from the CNN layers. The expansive path was composed of a successive combination of convolutional neural networks and upsampling layers in order to reach the original spatial resolution see . To avoiH represents height, W width, D depth, and C the numbers of channels. The objective of this study is to segment multiple anatomical brain structures, predicting the corresponding pixel-wise label maps of size The architecture input is as follows: given an image MHSA) modules and Multi-Layer Perceptron (MLP) blocks, each preceded by a normalization layer. The MLP blocks use the RELU activation function with a regularization dropout layer.We used self-attention mechanisms via Transformers in the encoder path. This consists of successive MHSA modules in each transformer using a set of vectors named as query, key, and value vectors [softmax function to obtain the weights. The attention matrix inside each MHSA head is computed as:Q, K, and V are matrices representing a set of queries, keys, and values, respectively.The attention mechanism was computed in parallel inside each of the heads of the  vectors . The queMHSA module. This can be described as:Finally, the results of each head were concatenated and linearly projected into a matrix sequence at the end of the In the proposed architecture, we used the extracted feature maps from previous convolutional layers as the input of the first transformer layer, using a trainable linear projection. Indeed, we reshaped the feature maps Segmentation of brain structures is a highly imbalanced problem due to the significant differences in size in the structures, presenting greater availability of information in the image for those of greater size. Even the size difference between the structures and the background is usually significant. Therefore, multiple loss functions and weighting strategies for loss functions were proposed for improving imbalanced brain structure segmentation . In the S is the total number of segmentation classes, Dice Loss (DL) has its origin in the Dice Similarity Coefficient (DSC), which is widely used as a metric for computer vision segmentation to calculate the similarity between two images. Later, in , it was FL) is a variation in Binary Cross-Entropy that works better with highly imbalanced datasets. It down-weights the contribution of easy examples and mostly focuses on the hard ones. It can be described as follows:Meanwhile, Focal Loss  and 1 (perfect segmentation).In order to evaluate the performance of the proposed segmentation method, the ground truth and model prediction from the MRIs were compared. The selected metric for this comparison is the precision is the accuracy of positive predictions while recall is the ratio of positive elements that were predicted correctly by the segmentation model. These metrics can be expressed as:FP the number of false positive predictions and Consequently, IoU) metric is included in this study as an evaluation metric for specific structures. It is useful for comparing similarities between two shapes Also, the Intersection over Union  architecture , we set We quantitatively and visually evaluated the performance of brain structures segmentation.The combination of these two loss functions helped us to alleviate the imbalance problem in the segmentation of anatomical brain structures and encourages the correct segmentation of tissue boundaries. Indeed, the use of class weights while training deep neural network architecture was necessary due to the large number of small structures the brain has compared with the total number of brain voxels. In order to calculate the class weights, we used the median frequency balancing algorithm, which is formulated as follows:A graphic example of the segmentation result of the proposed deep neural network architecture can be seen in Although to achieve the training process of the deep neural network architecture the brain is divided into blocks of equal size, the results show that the segmented structures maintain spatial coherence and can recover its representative organic form as can be seen in a 3D visual representation shown in For this experiment, we set four successive layers of transformers with patch resolution blocks of al U-Net -based arWe experimented with patch resolution sizes related to the transformer layers. As was initially observed in , patch rures see .At present, the majority of the proposed deep neural network architectures for brain segmentation using Transformers are oriented toward the segmentation of brain tumors. Therefore, it is highly difficult to have a fair comparison since these models were oriented to the segmentation of one class, and not for multiple classes excluding background. Because of this, we implemented the 3D U-Net architecture, using it as our baseline, with an identical experimental setup. This comparison using the Dice score and the Wilcoxon signed-rank test can be seen in The time needed to perform the segmentation by this architecture and the comparison with other deep learning models is shown in This study presents a deep learning-based model for the segmentation of 37 brain structures using transformer models. This network was trained with the manually annotated dataset Mingboggle-101, which contains 101 MRIs with its respective segmentation files processed using the Desikan-Killiany-Tourville (DKT) protocol . In the Our architecture includes self-attention modules to strengthen the connection between the encoding and decoding phases based on convolutional neural networks. The capabilities of self-attention modules add to the model the possibility of retaining features across voxels in the input patches of the model. Unlike 2D-based models, the 3D architecture can find voxel relationships in the three different planes, maximizing the use of the spatial nature implicit in MRI.In addition, the results of the proposed segmentation model show that the quality metrics have a wide range of values. For the Dice Score, for example, the values vary from 0.32 to values of 0.93, showing low-quality segmentations for some structures. We find that the lower values tend to be related to structures with smaller volumes. The geometry at the edges of the structures is a factor that we consider influences the quality of the segmentation. Structures with borders of highly variable geometry tend to have segmentations with more error. Simpler edge structures generally result in more stable quality segmentations. Our intuition in this regard is that this is due, mainly, to the class imbalance problem and lack of enough data to train the model for those structures specifically. For this reason, it is important in the future to explore other methods that allow addressing this problem, for example, improving the calculation of the weights of the classes used in the loss functions similar to what is performed in , or usinThe patch resolution size is a determining factor to obtain finer details at the edges of segmentations. The experiments show an inverse relationship with size, that is, the smaller the patch size, the more detailed segmentation is obtained at the edges; the larger the patch size, the segmentations tend to be less detailed. It must be considered not all structures have geometrically complex edges, there are structures with simpler geometrics. Therefore, a trade-off between the computational cost of reducing the patch size and the more detailed segmentation requirements must be considered. Given the 3D representation used in this study and the memory requirements, it was not possible to explore values smaller than 8 × 8 × 8.The results show that our method uses less time for segmentation with a Mean Dice Score similar to those found in the state of the art. Additionally, the segmentation of more than 25 brain structures into a 3D representation is a difficult task that has only been reported by a few groups of authors  due to cOur method still has deficiencies related to the variation in the segmentation quality for different structures. Class imbalance, as well as the broad geometric nature of the edges are factors for which our method is still sensitive. The number of segmented structures is also a limitation, it is desirable to be able to segment a greater number of structures, especially compared with 2D-based approaches.In future work, we will explore the existing computational and memory limitations in our proposed architecture with a high focus on the transformer layers to see whether a different tokenization of the patched feature maps can improve its performance and segment more brain structures.


In this paper, we conduct an in-depth study and analysis of the automatic image processing algorithm based on a multimodal Recurrent Neural Network (m-RNN) for light environment optimization. By analyzing the structure of m-RNN and combining the current research frontiers of image processing and natural language processing, we find out the problem of the ineffectiveness of m-RNN for some image generation descriptions, starting from both the image feature extraction part and text sequence data processing. Unlike traditional image automatic processing algorithms, this algorithm does not need to add complex rules manually. Still, it evaluates and filters through the training image collection and finally generates image automatic processing models by m-RNN. An image semantic segmentation algorithm is proposed based on multimodal attention and adaptive feature fusion. The main idea of the algorithm is to combine adaptive and feature fusion and then introduce data enhancement for small-scale multimodal light environment datasets by extracting the importance between images through multimodal attention. The model proposed in this paper can span the semantic differences of different modalities and construct feature relationships between different modalities to achieve an inferable, interpretable, and scalable feature representation of multimodal data. The automatic processing of light environment images using multimodal neural networks based on traditional algorithms eliminates manual processing and greatly reduces the time and effort of image processing. The continuous progress and development of Internet technology have brought an unprecedented amount of data to human beings, and such a large amount of data must have hidden potential values. The popularity of various hardware terminals also allows mining the matter in a large amount of data, and these changes have promoted machine learning research. The principles related to machine learning are mainly penetrated in two fields: computer vision (CV) and natural language processing (NLP). Computer vision focuses on making computers “see” the world the way humans do, that is, making them understand images. Nowadays, computer vision has become inseparable from human life, such as cell phones scanning QR codes, face recognition, video surveillance, and processing remote sensing images. Natural language processing enables computers to communicate with humans through human languages, such as using machines for language translation and sentiment analysis of website comments. Developments in these two fields have been able to assist humans in various tasks and are still being refined to be used for more complex problems in the future. The increase in the amount of data makes the data more and more complex, and single information such as image, language, and sound is no longer separated. Still, more often, they are fused, so the study of multimodal data is gradually becoming a hot research topic for researchers. For artificial intelligence to better understand the world, it is necessary to improve the machine's ability to understand different forms of information.Multimodal data covers various data types that people can access, including images, text, sound, and sensors. These data are not only naturally occurring but also abstract data types evolved by human beings in the long-term cognitive process, so, for multimodal data, each piece of modal data not only has a vast difference in the storage form but also has essential differences in the abstract semantic features. Therefore, for multimodal data, each piece of modal data has not only a huge difference in storage form but also a fundamental difference in abstract semantic features. Therefore, how to learn multimodal data features has been an essential topic of research in the field of data mining. Deep learning has proved its powerful feature extraction ability in several AI-related applications, so deep understanding has a vital role in feature learning in each modality, such as image feature extraction using a convolutional neural network to achieve image classification and speech feature extraction using a Recurrent Neural Network to acquire speech and text conversion. The ability of deep learning to learn features of a single modality is relatively mature. However, the modal components extracted by traditional deep learning feature extractors are uninterpretable, resulting in that the parts of multiple modalities cannot be well correlated. In addition, the features obtained by deep learning feature extraction are only the feature distribution of a large amount of data. Still, they cannot correspond with external knowledge and knowledge modeling, so deep learning cannot satisfy the relationship modeling and feature understanding between different modalities. For this reason, a more comprehensive multimodal feature learning model that can span semantic differences of other modalities and build feature relationships between different modalities is needed to seek a more comprehensive multimodal data referenceable, interpretable, and scalable feature representation. This paper proposes an automatic image processing algorithm for light environment optimization based on the multimodal neural network model. Based on the traditional algorithm, the multimodal neural network is used to automatically process the light environment image, eliminating the need for manual processing. It greatly shortens the time of image processing and saves time and effort.The structure of this paper is as follows.The first chapter is the introduction. This chapter summarizes the research background and significance of automatic image segmentation, research status, and existing problems and finally gives the research content, innovation, and structural arrangement of this paper.The second chapter is related work. To provide a theoretical basis for subsequent algorithms based on fully convolutional neural networks, this chapter mainly introduces image segmentation related technologies, fully convolutional neural networks, and deep learning related theories.The third chapter is the multimodal neural network model design. This chapter mainly introduces an improved automatic segmentation method of human images, which realizes the automatic segmentation of human images. In addition, this chapter locates the position of the character, assists the character segmentation network to segment the image of the character more accurately, and conducts experimental analysis and result evaluation of the algorithm.The fourth chapter realizes the automatic image processing algorithm for optimization of light environment. This chapter mainly introduces the further segmentation of the character edge region based on the character segmentation network in the third chapter and proposes an improved edge segmentation algorithm, which is based on the closed demapping algorithm, and adds a global color space model and dynamic local optimization. Finally, the algorithm is experimentally analyzed.The fifth chapter is the analysis of the results. The performance test of the multimodal neural network model and the simulation experiment of the light environment optimization image automatic processing algorithm are carried out.The sixth chapter presents the conclusions. This chapter summarizes the research results of this paper, points out the deficiencies, and looks forward to future work.The MCP artificial neuron model emerged, which was the first model to apply the idea of neural networks. The model uses a computer to simulate the neuronal transmission and response of the human brain and divides the complex neuron into three parts: the input part is multiplied with values based on weights, the middle of the neuron is a linear combination of the output values of the upper part, and finally the activation process is at the end of the simulated neuron . The perThreshold-Based Segmentation Method. Among the many image segmentation methods, the most classic and vibrant application scenario is the threshold segmentation method, mainly because of its simple algorithm principle, easy implementation, stable performance, and high efficiency [ficiency . Threshoficiency . Therefoficiency . Noda K.ficiency . The nulficiency .The continuous research on deep neural network algorithms has driven the implementation of deep learning in practical applications. Deep understanding has made remarkable development in the traditional computer vision image processing field and the emerging areas such as natural language processing and speech recognition . The focConvolutional neural networks (CNNs) are a class of feedforward neural networks that include convolutional computation and have a deep structure and are one of the representative deep learning algorithms. Convolutional neural networks are also known as “Shift-Invariant Artificial Neural Networks (SIANN)” because of their symbolic learning capability and their ability to classify input information shift-invariantly according to their hierarchical structure.Graph convolutional neural networks are essential for graph structure data feature extraction. In multimodal feature learning graphs, neural networks are applicable not only to modeling topological relationship graphs within each modality but also to modeling topological relationships between multiple modalities. The graph neural network based on spectral analysis is one of the most common neural graph networks. Its central idea is the propagation of features of adjacent nodes, where the general expression for the propagation of feature messages can be expressed as follows:lth layer of the graph neural network is represented as hil+1, g(∗) is the message mapping function, which is used to construct the message feature, and N(i) represents the set of adjacent nodes of the node. AGGREGATE means to aggregate the messages spread from different adjacent nodes, which can use cascade, maximum pooling, average pooling, and other functions, and σ means nonlinear activation function.In , the feaA graph convolutional neural network first requires the construction of a relational topology graph. Then the features of each node of the graph network are propagated through the adjacency relationship using the feature propagation algorithm. Finally, interfeature relationship extraction is performed. The feature representation output from the graph convolutional neural network is feature-extracted from the relational topological graph by pooling or cascading and so forth, to obtain abstract representations of the graph structure data at different levels . The gram images with n classes of social tags, and each print will correspond to several types of labels. The labeling relationship between m images X={xi}i=1m and n classes of social tags C={Tj}j=1m forms a binary matrix Y ∈ {0,1}m×n. The element yij ∈ {0,1} in the i-th row and j-th column of the matrix indicates whether the i-th image Xi is labeled Tj; when yij=1, it indicates that the user has marked the i-th image XiTj, and a zero indicates that it is not labeled. The social image relabeling algorithm aims to find an ideal set of labeling matrices Z ∈ {0,1}m×n that can correct the deficiencies of the original labeling matrix Y and accurately describe the image information. Matrix decomposition is a standard solution to this problem. It decomposes the social label matrix Y into two low-rank matrices Pi ∈ Rm×t, Qn×t such that Y ≈ PQt. The element of the i-th row in PPi ∈ Rt is the r-dimensional hidden feature of the image Xi. Classical matrix factorization models train the model using the L2 loss function, which implicitly assumes that the noise matrix E follows a Gaussian distribution.In the social image relabeling problem, the image dataset contains qi ∈ Rt in the j-th row of Q is the r-dimensional hidden feature of the label Tj. There are various matrix decomposition methods for solving P and Q in different ways for different applications. The most basic form is the singular value decomposition (SVD).Similarly, the element Y and the ideal labeling matrix Z is the noise E present in the social labeling. The perfect labeling matrix Z can be deduced backward if the noise E can be accurately modeled. The classical matrix decomposition model uses the L2 loss function to train the model, implicitly assuming that the noise matrix E obeys the Gaussian distribution. However, Gaussian distribution is not necessarily the most appropriate noise assumption, and there may be better probability distributions to obtain better results for image relabeling [t-distribution, and logistic distribution, and exploring the effect of social labeling noise, this section proposes the Corsi matrix decomposition method to generate better image annotation results by exploiting the robustness and excellent product of Corsi distribution in fitting social labeling noise. This section first derives the general probability matrix decomposition model, introduces five probability distribution assumptions, derives their formal representation, and introduces the distribution characteristics and finally introduces the Corsi matrix decomposition model in detail and verifies its effectiveness with experiments.The difference between the observation matrix labeling . By compThe semantic segmentation based on RGB-D multimodal images is mainly reflected in utilizing RGB images and Depth images. Nowadays, there are three common ways to combine them. The first is a single-branch structure, which directly stacks RGB and Depth images' channel information representation. It then uses them as inputs to the network to simultaneously extract feature information . The netDue to the difference in the information contained in Depth image and color image, Multimodal Complementary Attention (MCA) is proposed to perform feature extraction on any pixel point by first selecting the features of the two feature maps of RGB-D and assigning more significant weight to the essential elements that contribute more, and vice versa. A smaller weight is given. Since the feature points of the RGB image and Depth image are aligned one by one, the matrix of the two feature maps after weighting is summed. Finally, the fused image is added to the RGB-D fusion modality, which can supplement the general information of the RGB-D multimodal picture. As shown in The shallow features of deep convolutional neural networks mainly extract more abstract detail representations, while the deep parts are represented as factual semantic information. In the external feature fusion in the coding stage, this paper uses the MCA module to perform weighted complimentary summation operations on the multimodal features. In contrast, for the modular feature fusion in the middle transition between coding and decoding, it is necessary to enhance the acquisition of high-level semantic information by RGB-D multimodal features and to solve the problem of aligning the two original RGB-D image elements by the nonlinear mapping of successive convolution in the early coding stage. Based on this, Multimodal Global Attention (MGA) is proposed in this paper.With the rapid development of the industry, digital image technology has been widely used, and the field of video multimedia has developed rapidly. In the field of literature and art, the standard digital image technology is digital photography, post production, special effect production and editing, as well as game development, animation production, digital restoration of historical images, color restoration of black-and-white images, industrial design, etc., gradually forming a multimedia art culture dominated by computers. While digital image technology is developing rapidly and taking pictures is becoming more convenient, people are increasingly active in taking pictures of themselves or others. More photos of people are uploaded to the network . As the The overall process of the automated image processing system is shown in Qt Development Framework is a cross-platform development framework developed by a Norwegian software company, the famous KDE (Linux system popular software) developed for Qt. Qt provides a layer of abstraction to achieve the concept of a code running on various machines, supporting Windows, Linux, Mac, and embedded Linux. Qt for Python 12-741 is Qt's interface to the Qt runtime environment for Python, which allows you to use the Python language to call Qt's core libraries, UI graphical user interface, and so forth. Although Python initially supports cross-platform, its GUI support is poor. With Qt's GUI interface, Python's cross-platform interactivity is greatly enhanced.Qt are the graphical user interface, the web application interface, Webkit (which provides the HTML rendering engine), Qt Creator, and the signal and slot mechanism. Qt's signal and slot mechanism, instead of callback operations, when the user clicks a button, hovers over a component, drags and drops, and so forth, will be sent to the receiving slot in the form of a signal.The core modules of O, and background pixels are represented as class B. The best case for image segmentation is that the graph-cut boundary is at the edge of both the foreground and background. Let L={l1, l2, l3,…, lp}, where p is the number of pixels and lp ∈ {O, B}; that is, any pixel can be classified as foreground or background.Graph-cut algorithm is a commonly used image energy optimization algorithm and is a widely used image segmentation algorithm. Image segmentation can be considered a kind of image classification problem, where foreground pixels are denoted as class The number of samples collected in this project is only 64 cases. Due to the small number of pieces, if the commonly used convolutional neural networks (such as ResNet) are selected, the quality of the extracted image features may be affected by overfitting or model nonconvergence due to the large size of these models and too many parameters . TherefoThe NYU Depth V2 dataset contains 25,000 instances collected from the social photography site Flickr. Each image is tagged with 38 semantic concepts and some associated text labels and manually annotated using at least one of the 24 category tags. Our experiments selected 20,015 data points with no less than 20 text tags. The dimensionality of the feature vector for the images in this dataset is 150 dimensions, and a 50-dimensional feature vector represents the text. The NUS-WIDE dataset is a huge dataset composed of web images collected from reality, including 269,648 instances and images with associated text tags. There are 81 basic factual concepts available for manual annotation for retrieval evaluation [y = 0.3 and u = 0.1. In addition, the batch size is fixed at 128, and the algorithm can be run 500 times.For the MIRFLICKR-25K dataset, we randomly selected 10,000 instances as the training set. We used 2,000 instances of this dataset as the test set and the rest as the retrieval set for testing. We randomly sampled 10,500 data points for the NUS-WIDE dataset as the training set. We used 2,100 instances of this dataset as the test set and the rest as the retrieval set for testing. Alexie network has been pretrained on the ImageNet dataset and fine-tuned while training our model. In our experiments, we set As shown in For both NYU Depth V2 and SUN RGB-D datasets, the performance indexes obtained by the model with a one-layer relational graph convolution network are better, which indicates that these data have fewer high-order implicit relational features. In contrast, the use of multilayer relational graph convolution will lead to the overfitting of the model. Then the model performance decreases, while, for the Yelp2018 dataset, there are more layers of relational graph convolution. On the one hand, this is because the knowledge graph of this dataset has a low overlap rate of relational feature subgraphs, which cannot map each entity to a common relational space through first-order multirelational feature propagation. On the other hand, there are more users' implicit relational preferences in the dining dataset, so the more layers of relational graph convolution, the better the extraction of higher-order multirelational features and the better the model performance.If the model is split and deployed separately on two servers in a distributed environment and no alternate execution policy is used, the iteration cycle takes about 1.08 seconds. The waiting time includes the execution time of the fully connected module and the time to synchronize the data to the fully connected module. The convolutional module slows down the process by creating GPU idle time while waiting for the remote fully connected module to return data . If the F1, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Frechet Inception Distance (FID) Score are evaluation metrics to investigate the quality of image samples generated by CycleGAN-based data expansion method. The data samples expanded by CycleGAN outperform the data samples developed by random interception and noise perturbation methods in three evaluation metrics: PSNR, SSIM, and FID. Unlike PSNR and SSIM, FID uses the Inception network to extract image features. It then calculates the distance between the original image and the expanded image, so FID is evaluated from the perspective of convolutional features. The CycleGAN-based data expansion method performs complex operations but retains many essential features. The traditional data expansion method operates on the original image. Based on its simple algorithm principle, easy implementation, stable performance, and high efficiency, people choose the threshold segmentation method. The essence of threshold segmentation is to select different feature thresholds, compare them with the gray values of all pixels, and then use the results of the comparison to classify all pixels one by one, to achieve the effect of segmentation.In addition to the three evaluation metrics of automatic image annotation models, namely, average accuracy, average recall, and overall performance, In contrast, the CycleGAN-based data expansion method first extracts the image features. It then reconstructs the image based on the extracted image features, in which the detailed parts of the original image are ignored. In contrast, the key elements are retained, making the reconstructed generated image more conducive to training a high-performance convolutional neural network. This makes the reconstructed image more conducive to training a high-performance convolutional neural network. The experimental results of deformable convolution and Batch Normalization on network performance are shown in l decreases steeply, while the loss value of Scheme 3 decreases gently. The loss value of Scheme 3 is stable when the training reaches 110 batches, while the loss value of Scheme 1 still has a decreasing trend and reaches saturation when the activity goes 130 packs. The analysis of the above experimental results shows that reducing the number of parameters in the fully connected layer and adding Batch Normalization can make the training process more stable and significantly reduce the training time of the network.The experimental comparison in The segmentation results of RX-Net and RAX-Net proposed in this paper are closest to the actual labels. This demonstrates the effectiveness of the network structure, and the problem of semantic conflict can be effectively avoided by the multifeature extraction structure for multimodal images separately. The segmentation result of RX-Net is slightly inferior to that of RAX-Net, which is mainly because the noise in the picture makes the network misjudge the class of pixels. With the addition of the multiscale attention module, the segmentation results are significantly improved. This indicates that the multiscale attention module combined with multiscale features can provide a more comprehensive understanding of brain tumors' global and local information and enhance the weight of pixels in task-related regions based on this information. This also enables the network to determine the category attribution of pixels and suppress the interference caused by noise to the segmentation task more accurately.Therefore, how to learn multimodal data features has always been an important research topic in the field of data mining. In many artificial intelligence-related applications, deep learning has proven its powerful feature extraction ability, so deep learning plays a very important role in feature learning in various modalities. The Receiver Operating Characteristic Curve (ROC) is a tool used to measure the nonequilibrium nature of classification. The ROC Curve and the Area Under the Curve (AUC) are often used to evaluate the merits of binary classifiers. The ROC curves and AUC of different segmentation methods are shown in Recommendation model training time is an essential indicator of model iteration and efficiency of new data used in recommendation systems. Since the iterative update of users' preferences and products is time-sensitive, it is essential to speed up model training to improve the recommendation accuracy of the model in practical applications, and having a high model training efficiency can improve the mining ability of users' short-term preferences and thus enhance user experience. Since the model training time is not only related to the computational complexity of the model but also related to the convergence time of the model, in general, the relational graph convolutional network takes more time than the graph convolutional network, so the model with more layers of relational graph convolution is chosen to have a longer model training time compared to the model with fewer layers.The image quality after processing by the algorithm in this paper has improved better, as shown in In this paper, after processing the images with uneven illumination, the subjective visual effect shows that the equalization algorithm results in severe color distortion and overenhancement. The Gamma correction algorithm is better than the histogram equalization algorithm. Still, the shadow of the face in the image is not effectively handled, and the Gamma function is not adaptive enough in the value method. The algorithm in this paper deals with the problem of uneven illumination in the image effectively and has good color retention of the image and can correct the unevenly illuminated vision adaptively. To further compare the processing effects of different algorithms, this paper uses objective indicators such as standard deviation, average gradient, and information entropy to measure different algorithms. The standard deviation reflects the contrast characteristics of the image, and the average rise is an essential issue for image sharpness. An image preprocessing optimization algorithm is designed and implemented based on adaptive Gamma correction for complex lighting environments.This paper designs and implements a multimodal neural network-based automatic image processing algorithm for image optimization in light environments with uneven illumination. Simulation experiments show that the algorithm in this paper effectively enhances the contrast, sharpness, and information of images in a fair environment and has good use in attenuating the effect of a complex light environment on the image being accurate. By conducting experiments on NYU Depth V2 and SUN RGB-D datasets, the model proposed in this paper has high automatic processing accuracy and shows some advantages in objective evaluation. Most multimodal neural network-based image segmentation methods require a large amount of labeled data for supervised learning. However, supervised learning methods will be brutal to accomplish the corresponding tasks for images where it is difficult to collect a large amount of labeled data. Due to the limited ability and time of the author, the automatic segmentation of human images and the automatic processing system of human images proposed in this paper still have many areas worthy of optimization, and there is still a lot of research space and development work in the future. For the automatic segmentation of human images, it is necessary to optimize the existing automatic segmentation system of human images, mainly from two perspectives: one is to continue to improve the segmentation accuracy, and the other is to speed up the segmentation speed and reduce the amount of segmentation calculations.


Mental health is one of the most important contributors to the global burden of disease in children and adolescents. Mental health conditions are associated with lower quality of life in adulthood. These conditions have an early onset and typically first occur in childhood. However, little is known about how these conditions are related to service utilization before the initial diagnosis, or about the significance of the diagnosis on later service utilization. To study this, register data on 5-15-year-old children in the city of Oulu, Finland, covering the years 2013–2018 were used. To identify the association of mental health and behavioral conditions with service utilization, counterfactuals were constructed from children who were similarly diagnosed three years later. Event study regressions on several health care and service utilization outcomes were estimated. The findings showed that primary and specialized health care utilization increased sharply before the initial diagnosis and peaked during the time of diagnosis. Primary care utilization started decreasing slowly after, while specialized health care utilization remained high for two years after the diagnosis. Probability of visiting a mental health professional, use of rehabilitation services, psychiatric medication, and utilization of child protection services increased significantly after the diagnosis. The results highlighted the importance of outpatient health care in detecting and treating the conditions in children. In a fragmented system, knowledge on utilization trajectories in different services may be of help in allocating resources to improve the health of these children. Mental health is one of the most important contributors to the global burden on disease in children and adolescents , 2. MentIn comparison to non-mentally ill children, mental health conditions are associated with higher and recurrent utilization of health care , 16–18. Previous literature has focused on service utilization before the diagnosis , 28 or aIn contrast to the previous literature, I constructed the control group from children who received the same mental health diagnoses but three years later. The idea behind this was to overcome the challenge of selection bias and unobserved heterogeneity , 35. In The objectives of this study were as follows. First, to analyze the association of mental health and behavioral conditions with pre-diagnosis health care and service utilization. Second, to analyze how the trajectory of service utilization changed in a two-year follow-up and what was the association of the initial diagnosis. Finally, comparisons of the composition of the pre- and post-diagnosis health care service utilization were made to observe shifts between visits related to mental health and other visits.The next section presents the Finnish institutional setting, the data, and the empirical approach.Finland has a universal health care system . Public Child health clinics monitor and examine the physical and mental health of children under school age. School health care is a health care service for primary school students, and is typically located in schools or nearby locations. Child health clinic and school health care services are provided free of charge. Municipalities are required to organize regular health checks for children . At leasThe Finnish private health care sector is market-based. Private health care is mainly financed through out-of-pocket payments and private voluntary health insurance . A smallRehabilitation services can be provided for mental health and behavioral conditions. In the context of children and adolescents, these services typically include rehabilitation, adaption training courses tailored for specific mental health diagnoses, and rehabilitative psychotherapy. Rehabilitation services are provided by the Social Insurance Institution of Finland along with health centers and hospitals . Most ofIn severe mental and behavioral disorders, psychiatric medicines can be used as one element of the treatment . PrescriIn Finland, a child may be placed with child protection services if he or she is neglected at home or the child’s own behavior endangers his or her health or development . MunicipData were collected from several registers for the population aged 5–15 of the city of Oulu, Finland, for the period 2013–2018. With an approximate population of 200,000, Oulu is the fifth largest city in Finland. Pseudonymized individual identifiers were used to link data from different registers.Public primary care visits were gathered from two separate registers: the register of the city of Oulu and the Register of Primary Health Care Visits, which is maintained by the Finnish Institute for Health and Welfare. These registers contained visits to child health care, school health care, and the regular primary care available at health centers. Information included the date of the visit, specialty of the health care professional, diagnosis codes, and type of service . These two sets of register data covered much of the same ground but also complemented each other in terms of missing information in the variables. For example, in rare cases, one register may contain the diagnosis given during the visit while the other register may not. Also, a visit may include several inconsistently recorded events, which made it unreliable to measure the number of separate visits during one day with the same service provider. For these reasons, I limited the number visits to maximum of one per day per person.Data on public specialized care visits were gathered from the Care Register of Health Care, maintained by the Finnish Institute for Health and Welfare. This register contained data on inpatient and outpatient specialized health care services. Information in the data included the date of the visit, diagnosis codes, specialty of the health care professional, and service type of the provider. I used the service type to separately identify on-call duty visits. I used it also to identify visits that involved a scheduled appointment and daytime hospital care. I identified distinct visits with separate contact days.Data on private health care visits came from the registers of the Social Insurance Institution of Finland. These data included visits and procedures reimbursed by the NHI scheme. Additionally, the data record the date of the visit and specialty of the health care professional. Similarly to the above datasets, I identified separate visits using distinct dates.In all of the aforementioned registers, I separately identified visits to mental health professionals see . SpecialThe number of health care visits  were measured at child-quarter level. Visits to mental health professionals were measured with a binary variable. The binary variable indicated whether a child visited a mental health professional during a given quarter or not.In Finland, only physicians can diagnose mental health conditions. Information on mental health diagnoses came from public primary care and public inpatient and outpatient specialized care. The private health care visits were not used as this register did not record diagnoses. Public primary care visits in both the Register of the city of Oulu and the register of Primary Health Care Visits had diagnoses recorded separately in ICPC-2  coding and ICD10  coding. ICPC-2 is a coding system dedicated to classifying diagnoses in primary care, and, in the public primary care data, ICPC-2 diagnoses were more consistently recorded than ICD10 diagnoses. The specialized health care data used ICD10 coding. In the ICPC-2 coding, I included all primary diagnoses in the “P” category . In the ICD10 coding, I included all primary diagnoses in the “F” category .Data on the rehabilitation covered by the NHI were drawn from the registers of the Social Insurance Institution of Finland. The rehabilitation services provided by the Social Insurance Institution include, for example, rehabilitative psychotherapy and intensive medical rehabilitation. The data identified the start and end points of the service. Rehabilitation can be partly embedded in the primary care data, but I did not identify these separately from the primary care visits. Rehabilitation services are also provided directly by the city of Oulu. However, these services were outsourced to a private provider and these services were not part of the NHI scheme so I did not have access to these data. NHI covered rehabilitation utilization was measured with a binary variable. This variable indicated whether a child utilized this service during a given quarter or not.Data on psychiatric medication purchases originated from the prescription register maintained by Social Insurance Institution of Finland. The register include all prescription purchases from pharmacies that were reimbursed by the NHI. The data include the date of prescribing and the ATC code. In the classification of psychiatric medication, I included depression medication, antipsychotics (excluding lithium), benzodiazepines, and benzodiazepine related drugs. See Child protection information originated from the social service register of the city of Oulu. The data included variables for the month of the service and the type of social service. I included information that was labeled as “child protection.” These services included both in-home and out-of-home care. Child protection service utilization was measured with a binary variable indicating whether a child utilized this service in a given quarter or not.Information on children’s age, sex, and place of residence was retrieved from the population register of the Social Insurance Institution of Finland. Age and municipality of residence were always recorded at the end of the statistical year.The Finnish National Board on Research does not require an ethical review statement for studies using only register-based data . The stuChildren with mental health conditions can differ from the rest of the population in many ways and their health care and other service utilization trajectories can follow different paths. As these children are not randomly allocated to treatment and control groups, direct comparison of children with and without these conditions likely yields biased estimates. To take into account this problem, I followed the previous literature , 42, 43,The general idea is to overcome the challenge of selection bias and unobserved heterogeneity. The research design exploited the timing of the first mental health or behavioral disorder related diagnosis among those who were diagnosed during the sample period. Counterfactuals to the diagnosed group were drawn from those who received the same diagnosis three years later. Specifying the control group in this way increases the plausibility that the groups are similar.Specifically, the treatment and control groups were formed from two waves: those who got the initial diagnosis in 2014 or 2015 (treatment) and those who got the initial diagnosis in 2017 or 2018 (control). For the control group, a “placebo” diagnosis was assigned to have been made three years earlier than the actual time of initial diagnosis. As such, the association of the conditions around the diagnosis was identified from the change in service utilization between the two groups over time. A three-year window was chosen to assure that the treatment and control groups remain comparable and to allow the outcomes to be tracked for a sufficient amount of time after diagnosis. Overall, service utilization was tracked for one year before and two years after diagnosis. Additionally, I used propensity score matching (PSM) to further balance the treatment and the control groups with respect to observable characteristics. Specifically, the age , sex and the initial primary diagnosis (at three digit level) were used in the matching process. By matching on the initial diagnosis, a concern that the results could be driven by potential differences in the initial diagnoses between the groups is alleviated. Nearest neighbor matching algorithm with a caliper width equal to 0.2 of the standard deviation of the logit of the propensity score was used , 45. Eacyit denotes the outcome for child i at time t. Time was measured at quarterly intervals. α is a constant, treatmenti is a binary variable that denotes whether a child belonged to the treatment group, γt is a vector of time fixed effects, Xit contain controls for age fixed effects and sex, and ϵit is the error term. Iτ(t = τ) are binary variables that indicate the quarter relative to the quarter of diagnosis. τ = Q0 is the quarter of diagnosis, with negative values indicating pre-diagnosis quarters and positive values post-diagnosis quarters. τ = −Q1 is the omitted quarter and thus the parameters are estimated relative to this period. δτ indicate the event study estimates for the control group. θτ are the parameters of interest and indicate the event study estimates for the treatment group. The parameter estimates θτ are recovered from the difference in the outcome between the treatment and control groups in the given quarter relative to the difference in their outcomes in the baseline quarter τ = −Q1. In other words, θτ indicate the differences in the outcome trajectories between the treatment and control groups.The estimation framework was an event study (dynamic difference-in-differences) specification:θτ comes from the assumption that, absent the onset of a mental health condition, the outcomes for the treatment and control groups would run parallel. The plausibility of this assumption is increased by limiting the comparison to children who got the diagnosis. Also, because of the three-year time window, there is one-year gap year between the last quarter of the post-diagnosis tracking (τ = Q7) and the period of actual diagnosis in the control group. This mitigates the concern that potential changes in outcomes in the control group just before their actual diagnoses would bias the identification of the post-diagnosis parameters of the treatment group. However, if the outcomes in the control group were affected more than one year prior to the actual diagnosis, the parameter estimates for the post-diagnosis periods could still be biased. To verify this, I also plotted the trends in the outcomes for the control group to see if there were any notable changes more than one year prior to their actual diagnosis.The identification for ϵit across time, standard errors were clustered at the child level [All outcomes were aggregated to a quarterly level. Poisson regression was used to estimate ld level .See The final sample consisted of 617 individual children in both the treatment group and the control group. Overall, the number of observations was 14,808 in the panel in the three-year follow-up period (12 quarters). This was the sample used in the regression estimations.θτ from Q1. The IRR of primary care visits showed an increasing trend in the pre-diagnosis period in the treatment group  had a mental health diagnosis attached to them.This study analyzed the association of mental health and behavioral conditions with health care and service utilization during the year before the initial diagnosis and two years after the diagnosis. The number of primary care visits increased prior to the initial diagnosis. Specialized health care visits increased slightly but statistically insignificantly prior to the initial diagnosis. Also, the probability to visit a mental health care professional increased in the pre-diagnosis period.Primary care visits peaked during the quarter of diagnosis and began to decrease after that, reverting back to the pre-diagnosis level two years after the diagnosis. In contrast, specialized health care visits remained higher in the post-diagnosis period. The probability to visit a mental health care professional followed a trajectory similar to specialized health care visits. The probabilities of utilizing NHI covered rehabilitation and psychiatric medication increased statistically significantly after the diagnosis. Also, the probability of child protection service utilization increased by approximately 20% compared to the mean level of the outcome.In primary care, the fraction of visits to mental health care services increased from 13% to approximately 30% from pre-diagnosis to post-diagnosis year. In specialized health care, the largest increase came from daytime hospital care, in which, in the context of children, the treatment is typically mental health related. Additionally, the fraction of specialized health care visits that included a mental health diagnosis was 63% in the post-diagnosis period.Studies that have analyzed specific conditions such as bipolar disorder , 30, schHealth care professionals working in primary care can play an important role in the treatment and referral of children to specialized care and mental health professionals . BecauseThe results also highlight the importance of specialized care in the treatment following the diagnosis. Most of the treatment in specialized care was received in an outpatient setting. The decrease in primary care visits and the increase in specialized health care visits following the diagnosis indicate a transition of the treatment from primary care to specialized health care. Also, the finding that 63% of the specialized care visits had a mental health or behavioral diagnosis attached to them after the diagnosis indicates continuity of treatment in specialized care. Additionally, the increase in the share of daytime hospital care, which includes intensified psychiatric treatment, provides further evidence of this.A confirmed diagnosis of a medical condition can be a precondition for receiving certain services. This likely explains the increased utilization of NHI covered rehabilitation and psychiatric medication after the diagnosis. Selection for rehabilitation is based on a physician’s referral and the final decision to grant rehabilitation is made by social insurance offices . The rehThe prevalence of psychiatric medication in children increases with age and is higher in adolescence , 21. TypPrevious studies have found a strong association between foster care and mental health problems in children –25. The The special contribution of this study to the literature was the use of an empirical approach designed to tackle the problem of selection bias and unobserved heterogeneity in the population of interest. This approach enabled to plausibly study the association of the initial diagnosis with service utilization trajectories. In addition, this study used individual-level data gathered from several registers that enabled the analysis of not only health care utilization but also other service utilization such as child protection services. Also, the health care data included school and child health care, which was vital in the context of child population.A strength of this study was the combined register data gathered and merged from several different administrative registers that presented both health care and other service utilization in the population. Compared to survey data, which may suffer from response bias, register-based data is more reliable in terms of missing information and selection into data . The datA strength in the empirical methodology was that it took into account selection bias by identifying the association of mental health separately from the difference in the timing of the initial diagnosis within the diagnosed population. The presumption was that children diagnosed later in the data were similar to children diagnosed earlier in the data. A naive comparison of the diagnosed population with the non-diagnosed population could yield biased estimates since the two populations are likely to differ in several unobservable confounders that cannot be controlled. These confounders, which could simultaneously affect mental health conditions, probability of diagnosis, and service utilization, may include individual health behaviors, the environment in which children are brought up, and comorbidities.However, because the control group received the diagnosis later, a concern may still be that some systematic differences remained between the groups. For example, if a later received diagnosis correlates with clinical severity, and this clinical severity then affects the outcome trends, it may introduce bias to the parameters of interest. Matching with respect to the initial diagnosis alleviates some of this concern. However, within-diagnosis severity differences could still remain because of the potential later onset of the condition in the control group.A drawback in the empirical methodology was the trade-off between the comparability of the treatment and control groups as well as the analysis horizon. The shorter the window between the early and later diagnosed is, the more similar the groups are likely to be. However, this directly also limits the length of the follow-up period after the event of interest. Thus, it is challenging to analyze very long-run associations with this method.Even with the unique data set used in this study, a limitation was the sample size of the diagnosed population. This ruled out the possibility of performing a heterogeneity analysis by finer sub-classifications of the diagnoses. The results could differ between more severe and less severe diagnoses. Also, this study analyzed trajectories only around the initial diagnosis, and any following mental health and behavioral diagnoses were not studied. The initial diagnosis may precede more complex mental health conditions , 30, leaFurthermore, the study population was identified based on the diagnosis. Mental health and behavioral conditions can be underdiagnosed in children to some extent –52. ThusThis study evaluated the association of mental health and behavioral conditions with health care and other service utilization before and after the initial diagnosis. Findings showed that primary care utilization increased sharply right before the initial diagnosis. Specialized health care utilization had a more minor increase prior to the diagnosis. Primary care utilization started to decrease towards the pre-diagnosis level shortly after, while specialized care utilization remained higher two years after the initial diagnosis. The probability to utilize mental health professionals, rehabilitation, psychiatric mediation, and child protection services increased significantly after the diagnosis.The role of outpatient care in the detection and treatment of mental health and behavioral conditions is crucial in children and adolescents. Recognizing the trajectories of service utilization for children with these conditions is essential and could help policy makers in targeting resources within health care and social care to improve children’s health outcomes.S1 File(PDF)Click here for additional data file.S2 File(PDF)Click here for additional data file.S3 File(PDF)Click here for additional data file.S4 File(PDF)Click here for additional data file.S5 File(PDF)Click here for additional data file.S6 File(PDF)Click here for additional data file.