When reconstructing a 3D object, it is difficult to obtain accurate 3D geometric information using a single camera. In order to capture detailed geometric information of a 3D object, it is inevitable to increase the number of cameras to capture the object. However, cameras need to be synchronized in order to simultaneously capture frames. If cameras are incorrectly synchronized, many artifacts are produced in the reconstructed 3D object. The RealSense RGB-D camera, which is commonly used for obtaining geometric information of a 3D object, provides synchronization modes to mitigate synchronization errors. However, the synchronization modes provided by theRealSense cameras can only sync depth cameras and have limitations in the number of cameras that can be synchronized using a single host due to the hardware issue of stable data transmission. Therefore, in this paper, we propose a novel synchronization method that synchronizes an arbitrary number of RealSense cameras by adjusting the number of hosts to support stable data transmission. Our method establishes a master–slave architecture in order to synchronize the system clocks of the hosts. While synchronizing the system clocks, delays that resulted from the process of synchronization were estimated so that the difference between the system clocks could be minimized. Through synchronization of the system clocks, cameras connected to the different hosts can be synchronized based on the timestamp of the data received by the hosts. Thus, our method synchronizes theRealSense cameras to simultaneously capture accurate 3D information of an object at a constant frame rate without dropping it. In the fields of computer vision and graphics, many studies have attempted to reconstruct 3D objects from a single image ,2,3,4,5.In contrast to the single-image method, the utilization of multiple sensors enables capturing the full 3D geometry of an object ,7,8,9. BFor example, Intel’s RealSense RGB-D camera, known for its cost-efficiency and mobility, is widely used in 3D applications ,13,14,15GenLock mode is a function whereby multiple sensors synchronize upon the rising or falling edge of a signal generated from an external trigger. With this feature, depth frames can be triggered from an external source to capture at almost arbitrary times or frequencies, within an allowable time and frequency window. However, GenLock mode provided by the RealSense drops the frame rate of each sensor by half . The redSimilar to GenLock mode, Slave mode only syncs the depth camera of RealSense cameras and syncs to a periodic signal generated from an external trigger. The difference between GenLock mode and Slave mode is that Slave mode allows cameras to start capturing frames without a trigger while GenLock only starts capturing frames when the signal is given. Moreover, unlike GenLock mode, which halves the frame rate, Slave mode enables the sensors to record at a designated frame rate. However, Slave mode faces a challenge in synchronizing multiple cameras. Due to the hardware issue of the limited USB bandwidth, the maximum number of devices that can be connected to a single host for stable data transmission is three . In ordetime synchronization delay and network latency delay. Having all hosts share a common system clock allows synchronization of RGB-D cameras connected to different hosts, allowing the reliable capture of numerous RealSense cameras simultaneously without a frame rate drop. In our experiments, 18 RealSense RGB-D cameras were used to validate the proposed method, demonstrating accurate and reliable synchronization of multiple RealSense cameras.To tackle the problems faced by all these synchronization modes, we propose a novel synchronization method, which allows the synchronization of multiple RealSense cameras each connected to a different host. Because our method is an extension of Full Slave mode, our method can synchronize both RGB cameras and depth cameras in RealSense cameras and capture frames without dropping the frame rate. Furthermore, our method increases the number of synchronizable cameras by increasing the number of hosts in the multiview system so that more accurate geometric information of the 3D object can be obtained. Synchronization of multiple hosts was performed via setting all hosts to have the same system clock using the master–slave architecture by introducing two kinds of delays between the hosts: Many affordable RGB-D cameras are widely available today, accelerating research related to 3D reconstruction ,21,22,23The Orbbec Astra series uses a structured light technique to estimate depth. In the structured light technique, patterns whose original shapes are known in advance are projected to a target object. The depth is estimated using geometric relationships between the original and deformed pattern shapes. However, when the object is captured using multiple structured-light-based cameras, the patterns projected to the target object from different cameras interrupt each other. Thus, the deformed pattern shape from one camera cannot be distinguished clearly, making it difficult to use multiple structured-light-based cameras to capture a single scene.The Microsoft Kinect Azure uses a time-of-flight (ToF) technique that estimates depth by measuring the round-trip time of light. The Kinect Azure device supports the highest RGB resolution among other comparison cameras. As a consequence of supporting the highest resolution, it requires an extra power supply in addition to the USB cable. Furthermore, as the ToF technique measures the depth using the signal reflected from the target object, the quality of the depth significantly decreases when multiple ToF cameras are used to capture the object simultaneously. The Intel RealSense L515 uses a light detection and ranging (LiDAR) technique, which is one of the ToF techniques. Although the depth can be estimated more accurately using the LiDAR technique compared to other techniques, it is difficult to apply multiple cameras to capture a single object simultaneously, similar to the structured light technique, as reported in .The Intel RealSense D400 series uses an active stereo technique to estimate depth, where a projector emits an unstructured pattern to add texture to the surface of the object while the depth is calculated by matching texture correspondences between images captured by two infrared (IR) cameras. Among other cameras in Firstly, it has been reported that the depth quality of a single camera can be improved using multiple active stereo cameras  since muSecondly, the Intel RealSense D400 series provides a 90 FPS capture speed, whereas other cameras support up to 30 FPS. This enables a moving object to be captured more reliably and accurately. Finally, the Intel RealSense D400 series provides the highest depth resolution of 1280 × 720. This is consistent with the report [To synchronize multiple hosts in a system, it is important to set all hosts to have the same system clock. To synchronize the system clocks of all hosts, using the Network Time Protocol (NTP) is a standard procedure in many applications because of its high availability and its ease of use. The NTP synchronizes system clocks among a set of distributed time servers and clients over the Internet ,30. Systsensor, frame, backend, and time-of-arrival timestamps. While both the sensor and frame timestamps are measured by the camera’s device clock, the backend and time-of-arrival timestamps are measured by the host’s system clock.Because our method aims to accurately synchronize a multihost camera system, the timestamps measured by each sensor and host play significant roles in synchronizing the multihost camera system. The four timestamps provided from the camera are the sensor timestamp is a timestamp that marks the middle of the camera exposure, and the frame timestamp specifies the time when the first packet is sent to a host. sensor and frame timestamps is almost consistent and that they are similar to each other. The back-end timestamp measures the time when the host copies the data packet from the USB controller to the OS kernel. From back-end timestamps drastically fluctuates. The time-of-arrival timestamp then measures the time when the RealSense SDK, librealsense, receives the data, and time-of-arrival timestamps does not fluctuate as much as in sensor and time-of-arrival timestamps of the three RealSense RGB-D cameras connected to the same host.Although the sensor timestamp has consistent intervals in sensor timestamps makes the synchronization difficult. On the other hand, the time-of-arrival timestamps of the three cameras, as illustrated in time-of-arrival timestamp is measured by one host, which receives the data from the three cameras. Since the time-of-arrival timestamps match one another, it is easier to determine whether synchronization has been correctly performed or not. However, there are some sudden jumps shown in time-of-arrival timestamps of the cameras to not match at certain frames. Thus, the regressed linear model between the sensor and time-of-arrival timestamps, which is called the global timestamp, was used to correct big jumps in the delay, as shown in sensor or time-of-arrival timestamps of sequential frames should be consistent as the sensor captures frames at a fixed frame rate. However, in some cases, the frame can be occasionally captured outside the periodic frame rate. Thus, in order to correct these outliers, the linear regression between the sensor and time-of-arrival timestamps was modeled. Let sensor and time-of-arrival timestamps. The regression of the global timestamp a and b are the coefficients of the linear regression, and n is the number of the previous frame for the linear regression, respectively. RealSense cameras connected to the same host can be synchronized via the regression of the global timestamp. The application of synchronization can be categorized into two scenarios: online and offline. Hence, we computed the linear regression in  for synchronization. In our setup, triggers were hierarchically installed. One main trigger KOTRON TG-16C generated periodic signals, and three subtriggers KOTRON TG-4C were set to bypass mode. Each subtrigger covered 6 RealSense cameras. Six desktops installed with the Microsoft Windows 10 operating system were connected to three cameras each in order to cover the high bandwidth requirements of the cameras. Among the 6 desktops, 1 of them is selected as the master host. The schematic diagram of the hardware installation is represented in All the evaluations and experiments in this paper were performed using 18 Intel RealSense D455 RGB-D cameras. All cameras were connected to external triggers (The resolution and frames per second (FPS) of the RGB-D camera were specified in pairs. We selected the depth and RGB streams according to the experimental situation. The RGB-D cameras were configured to cover 360To synchronize multiple RealSense RGB-D cameras, an external trigger was required to generate the signals at 1.8 V pulses per second (PPS). The RealSense RGB-D cameras could receive these signals through a 9-pin connector hidden inside a small latch. The sync signal was connected to pin 5, whereas pin 9 was the ground, shown in In GenLock mode, cameras do not start capturing a scene without the signal from an external trigger even though capturing is requested by the SDK. In other words, the camera’s exposure starts only when the signal from an external trigger is received. Due to this characteristic, all cameras in a multiview system can simultaneously start capturing the scene using GenLock mode. To properly use GenLock mode, an external trigger should be turned off first. Then, the start request is sent to all cameras by the SDK, and finally, the trigger is turned on and sends a signal that matches the frame rate.Conversely, RealSense’s Slave and Full Slave modes start capturing the scene regardless of the signal’s existence. With the signal from the trigger, the cameras capture sequences synchronously. In contrast, without the signal, the cameras are unsynchronized and capture frames at a designated frame rate. Therefore, in order to synchronize the cameras in Slave and Full Slave modes, the external trigger is turned on beforehand and periodically sends the signal at a frequency that matches the frame rate. While signals are periodically sent, we can obtain the synchronized sequences by requesting cameras to start and stop capturing of the scene using the SDK.global timestamps with synchronization during 10 s of recording. From Despite successful synchronization by GenLock, Slave, and Full Slave modes, the synchronization of cameras was prone to be unstable for the first 5 s in the RealSense camera system. global or sensor timestamps were used to find the simultaneously captured frames according to the proposed synchronization method. Assuming that the system times of the hosts were accurately synchronized, frames simultaneously arriving from different hosts had the same global timestamps. Thus, we could obtain the simultaneously captured frames by gathering each frame from all cameras that had the same global timestamps. The timestamps among cameras were different in practice. Therefore, we gathered frames whose timestamp differences were within a predefined threshold global timestamp set Algorithm 3: Simultaneous frame-gathering algorithmEach frame was captured by every camera with timestamps. Although the frames can be captured synchronously via the signal from an external trigger, the host does not receive image frames simultaneously from the RealSense cameras. It was even more difficult to determine if the images were captured simultaneously from multiple hosts. The sensor timestamps instead of global timestamps, allowing highly accurate synchronization of multiple cameras. In other words, although the devices had different sensor timestamps, the timestamps could be aligned with each other by subtracting each device’s timestamps by its first timestamp. Therefore, in GenLock mode, the aligned sensor timestamps of each camera were used to gather the frames. The aligned frames by GenLock mode were used to quantitatively evaluate if our method accurately synchronized the cameras.On the other hand, all the cameras are able to start capturing sequences at the same time in GenLock mode. This enabled us to use the global timestamps from the 18 RealSense cameras with 6 hosts. It clearly shows that the cameras connected to the same host could be correctly synchronized via the global timestamps, but large variations existed among the timestamps captured in different hosts. global timestamps by capturing sequences with the proposed synchronization method. It is shown that the proposed synchronization method significantly decreased the variations between hosts’ timestamps. The gathering method in Algorithm 3 matched the closest timestamps within the threshold in sequences of cameras. The lower variations in global timestamp was measured using the system clock of each host. To correctly evaluate the global timestamps of multiple hosts, the timestamps needed to be aligned with the absolute time, i.e., the ground-truth time, and then compared to each other. Therefore, in order to quantitatively evaluate the performance of our method, the ground-truth for synchronization was required. The ground-truth for synchronization was obtained by performing GenLock mode for each test.However, the lower variations did not confirm that the cameras were synchronized correctly because the sensor timestamps of each synchronized camera could be normalized by subtracting the first sensor timestamp of the sequence, and with these normalized sensor timestamps, the frames captured by the cameras could be correctly ordered. Based on the correct ordering of the frames, the accuracy of our synchronization method could be evaluated.Even though GenLock mode captures a sequence at half of the specified frame rate and synchronizes only depth cameras, it allows accurate synchronization by specifying the start time of the capturing sequences. The AMD), to quantitatively evaluate the synchronization of multiple RealSense cameras as:m is the number of frames and global timestamp of the ith frame, respectively. The AMD measures the average difference between the maximum and minimum global timestamp at each frame and, as a result, can observe the accuracy of synchronization at each frame. Because the AMD measures the biggest difference between the cameras’ global timestamps at a certain frame, the AMD can be considered as the multiview camera system’s average maximum error. Therefore, a multiview camera system with the maximum error less than an interval between frames  can be considered synchronized. Even though the AMD was useful in evaluating the performance of the multiview camera system as a whole, it was difficult to evaluate the correctness of the synchronization between the cameras.We defined a metric, the average maximum delay (RMDV), to evaluate the synchronization as:n is the number of sensors and RMDV computes the mean variation of the delay between the cameras at each frame. In order to minimize the effect of outliers, the median of cameras’ global timestamp at each frame was used as a reference instead of the mean to compute the RMDV. The RMDV computes differences between the median global timestamp and the other cameras’ global timestamps and uses them to accurately measure the correctness of the synchronization.Thus, we defined another metric, the root mean delay variation  with a 120 Hz display. The digital clock used in this experiment displayed the time in milliseconds in order to capture any delays between the cameras capturing the frame. Displaying time in milliseconds was appropriate for the qualitative result because the time interval between frames at 90 FPS is around 11.11 ms. While capturing a sequence, the digital clock was visible from 6 RealSense RGB-D cameras. Thus, 6 images from the RGB and depth cameras were presented for the qualitative result.The qualitative result was obtained by capturing a sequence of a person rotating in the middle of a multiview camera system with a digital clock in his hand. The digital clock was displayed on a tablet (The qualitative results obtained from GenLock mode are shown in The qualitative results gained from the RGB-D cameras synchronized with the proposed method and the NTP are shown in Three-dimensional reconstruction was the ultimate goal of the multiview camera system. Therefore, we evaluated the improvements of the proposed method on the reconstruction accuracy when capturing objects using multiple cameras with multiple hosts. The average pth maps  was usedFor the 3D reconstruction from the multiview depths, we first calibrated the depth sensors with the standard multicamera calibration method using a checkerboard . Then, uThe reprojection error was computed by projecting the vertices of the mesh to every depth map to match the corresponding points of the depth map to the 3D vertices. When calculating the reprojection error, the 3D vertices invisible to each depth camera were excluded, which could be determined by the Z-buffer test . Once thThe proposed method resolved the problems in the Slave and Full Slave synchronization modes of the RealSense devices by synchronizing multiple hosts’ times. In our experiments, the RealSense cameras were connected to the motherboard USB 3.0 interfaces of the hosts for communication without using an additional PCI-E extension card. It has been reported that the number of RealSense sensors that a host can support is up to three  without Furthermore, although the GenLock synchronization mode of the RealSense cameras drops the frame rate by half and cannot synchronize RGB cameras simultaneously with cameras, GenLock mode can be used for other cameras to accurately synchronize multiple RGB-D cameras without the frame rate loss, when all cameras start capturing at the same time. In other words, other depth cameras can be easily synchronized without the proposed synchronization method if using GenLock mode.However, in many practical applications, it is difficult to specify the starting point of capturing for each camera. As a result, the typical GenLock synchronization scheme cannot be used, but the proposed method can still be used to accurately synchronize multiple cameras with GenLock mode. In addition, our method does not limit the number of devices that can be synchronized by linearly increasing the number of hosts, allowing the multiview camera system to be largely scalable.global timestamps of all cameras. The system clocks were synchronized by estimating the delay occurring while synchronization takes place. The estimation of the delay minimized the offset between the master host’s system clock and the slave hosts’ system clocks. The result quantitatively and qualitatively showed that our method reduces the synchronization error between sensors significantly, enabling simultaneous capturing by numerous RealSense cameras without dropping the frame rate. We expect that our method will be helpful in various applications of computer vision related to 3D reconstruction [In this paper, we proposed a novel synchronization method for synchronizing RealSense RGB-D cameras in a multiview camera system. The proposed method tackles the limitation of RealSense camera synchronization, where multiple camera synchronizations using multiple hosts are possible with dropping the frame rate and depth cameras only. As the number of RealSense cameras to be used increases, more hosts are required to support a multiview camera system. To synchronize the RealSense cameras connected to different hosts, the hosts’ system clocks were synchronized in order to correctly regress the truction ,32.


Due to the complexity and high heterogeneity of the acute exacerbation of chronic obstructive pulmonary disease (AECOPD), the guidelines  is unable to fully guide the treatment of AECOPD.To provide a rapid treatment in line with the development of the AECOPD after admission. In this paper, we propose a multi-stage feature fusion (MSFF) framework combining machine learning to track the diseases deterioration risk of the AECOPD.First, we identify 408 AECOPD patients as the study population. Then, feature segment and fusion methods are applied to generate the phased data set. Finally, human studies are designed to evaluate the performance of the MSFF framework.The experimental results show that the proposed framework is potential to obtain the full-process tracking of deterioration risk for the AECOPD patients. The proposed MSFF framework achieves a higher overall accuracy average and F1 scores than the four physician groups i.e., IM, Surgery, Emergency, and ICU.The proposed MSFF model may serve as a useful disease tracking tool to estimate the deterioration risk at each stage, and finally achieve the disease monitoring and management for AECOPD patients. Chronic obstructive pulmonary disease (COPD), as a common disease characterized by an irreversible persistent airflow limitation, causes the decline in the quality of life of patients , 2. COPDIn recent years, Artificial Intelligence (AI) techniques have become potentially powerful tools for the diagnosis and management of diseases, mimicking and even aiding clinical decision-making of human physicians –15. For To improve the therapeutic effect of the AECOPD, many scholars study the disease based on artificial intelligence. Swaminatha et al. proposed a supervised Gradient Enhanced Random Forest (GERF) model after the collection of the clinical data from AECOPD patients. The prediction accuracy of GERF reaches 88% of that of clinicians . Wang etHowever, the timeliness of clinical data is the key to the real world research (RWD) . The negThe experimental results show that the proposed framework is potential to obtain the full-process tracking of deterioration risk for the AECOPD patients. The framework can also be extended to the deterioration risk monitoring for the other chronic diseases. The main contributions of this paper are summarized as follows: (1) A method based on machine learning is proposed to track the disease deterioration risk of the AECOPD with the generation time of clinical data. (2) We evaluate the proposed method by real data set from the Third Affiliated Hospital, Sun Yat-sen University.The remainder of this paper is organized as follows. Section We conduct the research under the guidance of the Third Affiliated Hospital, Sun Yat-sen University (TAHSYU) Institutional Review Board (IRB), protocol [2019]-02-334-01. 408 AECOPD patients are identified retrospectively by the International Classification of Diseases, Tenth Revision, Clinical Modification  from the respiratory unit database of TAHSYU, a major Chinese large-scale Medical Center. Data masking techniques are applied to the AECOPD patients before analysis.The distribution of AECOPD patients is shown in Table P. A patient in the collection can be defined as n clinical features, then k periods (phases or stages), so that the clinical features of k parts by periods, and then the clinic data of i produced at period phase? k. We use w ) generated at phase k of the patient.k can be represented as k stages as We define the data set of AECOPD patients as gi_kw as  to the rk, respectively.While all the patients’ time-based phased accumulating clinical data can be expressed by:f on the AEOCPD patient data set D through machine learning method. Then, the prediction of the staged deterioration risk for the AECOPD patient f denotes the machine learning model to be solved. k. f can be simple or complex model. The smaller the difference f from the data set Data modeling begins after the feature fusion. Assuming that there is a hypothesis D, segment the input D into K pieces, and define the cumulative data successively. Step 5 to step 11: for each phased data, construct the machine learning model successively and calculate the prediction results. Step 12: return the phased prediction results.We define the multi-stage feature fusion framework below. In this section, we propose the novel framework by combining the machine learning model and phased accumulating clinical data Fig. . The proWe carry out a study to compare the performance of MSFF framework with that of human physicians on the validation set of data set of k becomes bigger with the development of the treatment. For the convenience of experimental demonstration, we identify 4 segmentations and 40 features as the input of the MSFF framework , Support vector machin (SVM), K-Nearest Neighbor (KNN) and random forest (RF)) of the MSFF model, the overall accuracy and F1-score are used. The F1-measure and the overall accuracy (acc), Sensitive (sn), Specificity (sp), Matthews correlation coefficient (MCC) are the metrics with True Positives (TP), True Negative (TN),False Negatives (FN) and False Positives (FP). The receiver operating characteristic curve (ROC) are also employed to test the MSFF framework. Metrics are defined as follows:To guarantee the robustness of the MSFF framework, both cross-validation and independent test results are provided. Figure After a comprehensive comparison of various indicators, it can be found that the developed RF, PLS and SVM classifier in the MSFF framework performs better than KNN classifiers on the four stages, respectively. With the increase of data, the proposed MSFF framework can bring the clinicians the continuous risk observation window. Thus, the mining of the generation time in these clinical data is supportive of clinical decision-making.On a newly collected data, a contrast experiment between MSFF with RF classifier and clinical physicians on phased data MSFF is conceived to comprehensively integrate the multi-stage clinic data to forecast the exacerbation risks of AECOPD before they occur. Based on a comparison with junior physicians from Internal Medicine (IM), Surgery, Intensive Care Unit (ICU), and Emergency, the proposed MSFF framework obtains superior performance when forecasting the AECOPD exacerbation risk from real world data. There are several issues that require further discussion.K is determined  of AECOPD patients after discharge from hospital because of the poor follow-up compliance of the study subjects. Medical imaging was not used during data acquisition as we used the lung function to determine the severity level. In the future, we will establish multicenter contracts to obtain more AECOPD patient data. We hope that the model’s predictive power will be improved by more abundant and reliable real-world training data.To achieve the real-time monitoring of acute exacerbation disease such as AECOPD, we digs down the generation time of the clinical features, dynamically divides the clinical data into multiple stages, and utilizes the machine learning methods to perform the deterioration risk warning in stages, eventually achieves the real-time monitoring for the acute exacerbation diseases. The proposed MSFF framework is able to track the phased deterioration risk of AECOPD patients with real world data. Our model achieves a higher classification performance than the four junior physician groups. The data segmentation proposed in this paper conforms to the process of clinical diagnostic reasoning. With the increase of clinical information, the predictive performance of the proposed MSFF framework may be gradually improved. Further work we will investigate the natural language processing technology to dig out potentially valuable information from the electronic medical records, such as the patient’s past history, present medical history, course of illness, discharge summary and other text records .


Several studies have shown the importance of proper chewing and the effect of chewing speed on the human health in terms of caloric intake and even cognitive functions. This study aims at designing algorithms for determining the chew count from video recordings of subjects consuming food items. A novel algorithm based on image and signal processing techniques has been developed to continuously capture the area of interest from the video clips, determine facial landmarks, generate the chewing signal, and process the signal with two methods: low pass filter, and discrete wavelet decomposition. Peak detection was used to determine the chew count from the output of the processed chewing signal. The system was tested using recordings from 100 subjects at three different chewing speeds  without any constraints on gender, skin color, facial hair, or ambience. The low pass filter algorithm achieved the best mean absolute percentage error of 6.48%, 7.76%, and 8.38% for the slow, normal, and fast chewing speeds, respectively. The performance was also evaluated using the Bland-Altman plot, which showed that most of the points lie within the lines of agreement. However, the algorithm needs improvement for faster chewing, but it surpasses the performance of the relevant literature. This research provides a reliable and accurate method for determining the chew count. The proposed methods facilitate the study of the chewing behavior in natural settings without any cumbersome hardware that may affect the results. This work can facilitate research into chewing behavior while using smart devices. Chewing  is the action of crushing and grounding food by the teeth. It is an important process that represents the first step of digestion by which the surface area of the food is increased to allow for easy swallowing and efficient breakdown by enzymes. Healthy nutrition is affected by several factors related to chewing, including; food intake, chewing behavior, chewing time, chewing speed and the bolus size.Monitoring and study of the chewing process is important. Abnormal chewing behavior could be an indication of some ailments , which may reduce the chewing speed or the bolus size. Moreover, people suffering from binge eating disorder tend to consume large amounts of food in a short time and are subject to greater risk of high blood pressure and cardiovascular diseases . In addiWe record chewing video data from 100 subjects at three speeds .We use image processing techniques to isolate and extract the videos of the subject’s face away from artifacts.We extract signals corresponding to the various movements during the chewing action.We propose two algorithms to count the number of chews automatically based on Discrete Wavelet Decomposition and low pass filters.We achieve a low mean percentage error in automatically counting the number of chews.Manually counting chews by trained clinicians and the effort involved in studies enlisting even small number of subjects is large considering the number of chews per minute. The process is tedious, time consuming, and error prone. The objective of this paper is to automatically determine the chew count from video recordings of subject munching on food while using camera-equipped electronic devices. This research develops a method to automatically count the number of chews appearing in the video recording. The results from this work can facilitate greater research in chewing behavior and its relationship with human health. The contributions of this paper are as follows:The remainder of this paper is organized as follows: In Chewing is the process of grinding a large piece of food between the teeth to convert the food to small bolus that could be swallowed ,6. RecenThe relationship between chewing behavior and various health aspects is continuously being investigated in the literature.  showed tChewing has also been found to be beneficial to brain functions. Chen et al.  showed tTraditional methods used for determining the chew count were either manual or automatic  . Manual Automated methods employ a range of devices that vary in sophistication and cost. Some studies used Electromyography (EMG) to record the chew count of a small number of subjects , which is understood given that special electrodes, EMG device, and professional help are required to perform the recording ,20,21. IIn general, hardware-based methods may cause discomfort to child subjects and incur high cost in large-scale experiments. Additionally, remote or at a distance studies may not be possible if special procedures are required to fit the hardware. Cadavid et al.  used an The current study was approved by the institutional review board (IRB No. 29/11/2018) at King Abdullah University Hospital (KAUH) and the Deanship of Scientific Research at Jordan University of Science and Technology in Jordan.Written informed consent was sought and provided prior to the study commencement. For underage subjects, their parents filled the consent form, which needed to be signed if they voluntarily accepted their child’s participation. The research assistants received intensive training by the lead investigators on the data collection process, as well as the data entry. The information package included an information sheet describing the study purpose and procedure in details, the consent form (including consent to publication of images), and a parental/self-reporting questionnaire that contains demographics and other relevant information.The current study enrolled 100 randomly selected subjects. A total of 375 information packages were randomly distributed prior to data collection. Of those, 275 (73.3%) recipients refused to participate. The subjects included a mix of children and adults, with an age range of 6–76 years . Fifty-six of the subjects were children and 44 were adults, and 58 were males. There were no restrictions regarding skin color, facial hair, hairstyle, head cover, or wearing glasses .A Huawei Y7 Prime 2018 smartphone main camera was used for video recording. It is a 13 MP camera with 1080p@30fps resolution. The subjects were asked to face the camera and eat a crunchy food sample . Each subject recorded three one-minute clips corresponding to three speeds . There was no specific environment for the dataset collection, and no additional constrains were set during video recording. Videos were recorded in a variety of setups  and with different light intensities.Objective reference is required as a gold standard for performance evaluation. To this end, three annotators were trained by the principle investigators to count the number of chews in video recordings, and the training videos were not included in the dataset. Each annotator worked independently from all others and recorded the number of chews in each of the 300 video clips . The annotators were allowed to pause and rewind the videos for accurate counting.Upon completing the annotation, the reliability of the process was verified using Intra-class correlation coefficient (ICC) . Table 1The image is converted to gray scale, which reduces the overhead. However, once the face is detected, the location is marked in the colored image.The image is scanned to search for intensity differences that may represent facial features. This is done using boxes called Haar rectangles .These boEach box is represented by a matrix of values corresponding to the pixel color intensities in that box. The darker the pixel the closer the corresponding value to 1. A Feature is generated by the difference between the sum of pixel values in the dark region and the sum of pixel values in the light region.The previous calculations can cause high computational overhead because of the large number of pixels. Therefore, the process is adjusted to use an integral image . Each value, Equation . Figure Scanning the image using the rectangular boxes will generate a set of intensity values, which form the input to the classification process. The output of this step indicates whether or not a feature is likely to be part of the face. The Viola-Jones algorithm uses adaptive boosting (AdaBoost), which employs a weak learner constraint to select few features out of thousands of possible features. The algorithm training dataset contained 4960 annotated facial images as well as 9544 other images without faces .Cascaded or ensemble classification. This step further refines the classification process by attempting to discard the background regions by increasing the complexity of classifiers in cascade. The collective effect of the weak classifiers selects the best combination of features and their associated weights.The first step in the algorithm aims to detect the face of the subject. To this end, the Viola-Jones face detector was employed. The algorithm was chosen because it is fast and has high detection accuracy . It workThe Viola-Jones algorithm generates a bounding box around the face of the subject. However, the face as a whole is not useful by itself for chew counting. Thus, Kasemi and Sullivan landmark detector  was emplThe lower lip moves up and down during crushing the bolus in between the upper and lower jaws. Furthermore, the lower lip moves slightly to the left and right during the bolus motion in the mouth, but the motion of the lower lip decreases when the subject swallows. Moreover, the lower lip motion is undiscernible when the chewing speed is too slow and when the food texture is neither solid nor crispy. In addition, the separation between the two lips increases when the subject is taking a bite.The upper lip motion is unbeneficial for counting chews as it is undiscernible across video frames. This mainly due to its connection to the immobile maxilla.The corner points on the edge of the mouth move in an oval trajectory, which could be a result of smiling or other facial expressions. Thus, they were ignored.This procedure results in a Up and down during for crushing/chewing the food.Sideways during bolus motion across the mouth sides.A large downward movement for every food bite.Careful inspection of the chewing process revealed that most of the points responding to the chewing operation are located in the chin and jawline regions. Therefore, only 11 points in the chin and jawline were used, see x, y) and the reference point We define the up down mandible motion as one chew. To measure this motion, a reference point was required with the constraint that it is unaffected by the chewing motion, random movement, and may not be hidden during chewing. To this end, the upper left corner of the face bounding box was chosen as a reference for all movements. This box tracks the face throughout the recording and represents a fixed reference frame for the jawline points. The Euclidean distance (ED) was measured for each frame between every jawline point (Equation .(2)EDLow pass filter (LPF): a LPF was designed with a cut-off frequency of 1 Hz and a sampling rate of 30 Hz . It is aDiscrete wavelet decomposition (DWD): DWD is a discrete version of the continuous wavelet transform . It retaAs previously stated, the chewing signal carries some noise due to the subject’s movement, mandible motion, and other artefacts . We experiment with two signal processing methods to improve the signal usefulness, as follows:The output from either one of the two signal processing techniques  forms the basis for determining the number of chews. A peak detection algorithm was employed to detect the chewing markers. The algorithm works by finding every local maximum in the signal that is larger than the adjacent two neighboring points, where every peak represents one chew. The Minimum-Peak-Height (MPH) parameter for peak detection was set for LPF to half the average of all peak heights (PH), see Equation . For DWDThe MPH was set differently for the three chewing speed signals because it was observed that the mandible movement changes in response to different chewing speeds. The highest displacement occurred in the slow chewing speed signals. Thus, the chewing peaks were high in comparison to false peaks . On the other hand, the mandible displacement was small in the fast chewing speed signals, so more of the peaks need to be counted. N is the number of pixels in the image. The calculations are done within a small region of interest in the integral image. Moreover, the Haar features are computed in constant time [As presented earlier, the proposed work relies on software-based methods as opposed to hardware solutions . Sensing and counting hardware maybe invasive but it provides less computationally intensive option. However, the approach used in this paper is based upon well-established practical methods with linear time complexity. The Viola-Jones face detector runs in linear time ant time . The nexant time . Both thant time . The thiant time . The lasThe performance of the proposed methods was evaluated in terms of the absolute error (quations –9) to s to sAEThe Bland-Altman plot was used to measure the agreement between the proposed algorithms and the actual chew count as determine by each annotator. This is a graphical method that plots the difference between the calculated values and the gold standard values against the average of the two methods. Any two methods can be used interchangeably used if 95% of the data points are located within the limits of agreement, which are defined as the mean 1.96×SD .The work in this paper presents a method for the automatic counting of chewing from video recordings. The results from both the LPF and DWD approaches suggest that the proposed method can be used as an objective and accurate chewing counter. In comparison to the literature, the method was tested on a reasonably large number of subjects and chewing speeds.In both signal processing techniques, the algorithm was used to estimate chew counts in manually annotated chewing clips and was able to achieve a best AE, MAPE, and RMS of This study has several limitations. First, we did not experiment with different food types . Second, the gold standard depends on the annotators, who-although trained- are subject to mistakes and underestimation . It woulNonetheless, the proposed approach has several merits. First, no extra hardware is required for the deployment and usability of the counting algorithm. Once the system is installed, researchers who are interested in studying the chewing behavior of subjects  can use it easily. It can be used in natural everyday settings . Second, the study used a reasonably large number of subjects and investigated a wide range of chewing speeds. In comparison, the number of subjects in the relevant literature was less than 50 ,35. ThirChewing is an important process in the digestive system with much research dedicated to studying the effects of chew speed, chewing rate, and bolus size on the human health . In addition, it has been found that chewing speed is associated with cognitive functions.Recent proliferation of mobile smart devices, which are equipped with cameras and strong processing power, facilitated the development of many applications from a wide range of disciplines. Another aspect to consider is the health impacts of these devices, which are being used during everyday activities including eating. Thus, the work in this paper allows for the monitoring of the chewing behavior to enable researchers to further study human dietary habits while using smart devices.In this research, an algorithm was developed to count the number of chews from eating video recordings. The input is processed using two well-known and established methods  followed by a peak counting algorithm. Performance evaluation results greatly improved on the existing literature. Moreover, the system allows for the natural measurement without the need for expensive or uncomfortable hardware. We expect this work to enable further studies into eating and weight disorders, especially those connected to smart devices.


COVID-19 has sparked a debate on the vulnerability of densely populated cities. Some studies argue that high-density urban centers are more vulnerable to infectious diseases due to a higher chance of infection in crowded urban environments. Other studies, however, argue that connectivity rather than population density plays a more significant role in the spread of COVID-19. While several studies have examined the role of urban density and connectivity in Europe and the U.S., few studies have been conducted in Asian countries. This study aims to investigate the role of urban spatial structure on COVID-19 by comparing different measures of urban density and connectivity during the first eight months of the outbreak in Korea. Two measures of density were derived from the Korean census, and four measures of connectivity were computed using social network analysis of the Origin-Destination data from the 2020 Korea Transport Database. We fitted both OLS and negative binomial models to the number of confirmed COVID-19 patients and its infection rates at the county level, collected individually from regional government websites in Korea. Results show that both density and connectivity play an important role in the proliferation of the COVID-19 outbreak in Korea. However, we found that the connectivity measure, particularly a measure of network centrality, was a better indicator of COVID-19 proliferation than the density measures. Our findings imply that policies that take into account different types of connectivity between cities might be necessary to contain the outbreak in the early phase. Since the outbreak of COVID-19 in December 2020, the virus has rampaged across the globe, causing 2.6 million deaths at the time of writing. South Korea has recorded a cumulated total of 92,817 positive cases and 1642 deaths. From its inception in China, the outbreak of the virus proliferated through Europe, the U.S., and eventually across all continents, at which point the World Health Organization officially declared it to be a pandemic ,4.While large cities offer many benefits, tightly connected networks of goods and people around urban cores could accelerate the COVID-19 pandemic. Matthew and McDonald  noted thurbanization, such as New York, London, and Seoul, appear to be more susceptible to the proliferation of COVID-19. Previous studies have reported that infectious global diseases often leave behind a greater number of infections and a higher death toll urbanizatath toll  also notath toll  found thGiven these mixed findings, this study aimed to investigate the role of urban spatial structure on COVID-19 during the first eight months of the outbreak in Korea. Specifically, we focused on two factors related to the built environment: density and connectivity. By comparing different measures of density and connectivity, we sought to provide empirical evidence to determine which of the two measures are more related to the spread of COVID-19 in Korea. This research helps inform public health agencies and municipalities to find more effective measures of prevention and response against COVID-19 and other infectious disease threats.Globalization and urbanization have progressed rapidly over the past 100 years. Globalization, prompted by the development of transportation and communication technologies, has been recognized as a distinct shift in the spatial and temporal dimensions of social and economic life, a phenomenon known as “time-space compression” . This trSeveral high-profile infectious disease outbreaks occurred before the COVID-19 pandemic. For example, the 1918–1919 Influenza pandemic, known as the Spanish Flu, was the most severe pandemic in recent history, resulting in about 50 million deaths around the world. During this pandemic, the United States had implemented stringent social distancing measures, such as the closure of theaters, churches, schools, and saloons as well as the mandatory wearing of face masks . Since tIn both popular media and scholarly communities, there is an increasing debate around urban density and COVID-19 pandemic ,19. On tAs we are responding to the immediate COVID-19 crisis, this pandemic highlights the need to start thinking about how to rebuild our cities and neighborhoods against future shocks. For example, urban planners have been increasingly discussing alternative urban models, such as “Superblocks”, “Tactical urbanism”, or “15-min City” to overcome some of the major problems that have been exposed during the pandemic ,24,25,26Many empirical studies have attempted to assess the effects of urban factors on the spread of infectious disease, but the results have been mixed. In a study of the 1918 Spanish Influenza (H1N1), Garrett  assessedAnother important factor related to urban spatial structures concerns connectivity. In a study of the 1918 Spanish Influenza in England and Wales, Chowell  exploredWhile there is a growing body of research examining the impact of urban spatial structure on COVID-19 outcomes, empirical evidence is mixed and inconclusive. These mixed results may stem from the use of different measurements across the study. Especially, the results could differ by how the dependent variable is measured and calculated. For example, in research conducted by Hamidi et al. , when thMoreover, some measures of urban densities could be biased when the denominator includes undevelopable lands, such as mountain ranges and rivers. Hence, we should take caution while calculating population and employment densities by using administrative regions that only include inhabitable areas. The measure of connectivity could also affect study results. For example, Hamidi et al.  used a pIn assessing the impact of connectivity, the social network analysis has been used in the infectious diseases studies to identify geographic transmission hubs of the infection as well as geographic distribution of infected patients ,35. In aIn this study, we aimed to fill the gap in the literature by comparing different measures of urban density and connectivity during the first eight months of the outbreak in Korea. We make contributions to the existing literature on three perspectives. First, we used the actual density indicators that take into account only habitable lands. Second, we used the aggregated origination and destination (O/D) data for all travel modes to calculate the degree of connectivity, closeness centrality, betweenness centrality, and eigenvector centrality to gain insight into how the connectivity of the overall transportation network affected the spread of COVID-19. Finally, we tested the sensitivity of the results by using different measures of dependent variables. This study provides insights into multiple factors of urban spatial structure, reflecting the current debate at play on the vulnerability of densely populated cities.Our study area was the entire country of South Korea to empirically identify the main causes of the spatial proliferation of COVID-19. In response to the COVID-19 outbreak, the Korean government has adopted a stringent contact-tracing strategy, called the 3T framework (Testing-Tracing-Treatment), to control and reduce the number of COVID-19 cases . BecauseMoreover, private firms as well as public offices implemented flexible work schedules, such as working from home and flexible working hours, as well as encouraging their employees to hold virtual meetings in lieu of in-person meetings. A survey conducted by the Korea Enterprises Federation (KEF)  among thThanks to the Korean government’s rapid response and strict hygiene practices adopted by citizens and private firms, the infection rates have stayed well under control, with the rates ranging between 2 and 2.5% and even staying at 3.3% during the third spike in December 2020. This level is well below the 10% benchmark level that the World Health Organization recommends as adequate test-positivity levels . In addiThe spatial unit of analysis in our study was a city, a county, and a district level, which is a smaller geographic unit than counties in the U.S. ,46 and iOur main outcome variable was the accrued number of positive cases starting from 20 January 2020 until 17 September 2020, collected per county, which is Si/Gun/Gu. There was no official county-level COVID-19 data, so we manually compiled the data from websites of each city, county, and district. The actual government websites from which we sourced the original data can be found in the Of the variables that have the potential to affect the outcome variable, we chose indicators of density and connectivity as the explanatory variables. We calculated net density measures for the population and employment densities. Using the aggregated travel data with origination/destination(O/D) at the county level, social network theory was used to calculate the four different measures of connectivity. We further standardized the density and connectivity measures to enable robust comparison in the model results.The urbanization-related indicators of density used in this research were population and employment densities. Data for these variables were drawn from the Korea Statistical Information Service (KOSIS). In order to accurately determine the effects of urban densities, we used the concept of net density, which refers to the number of housing units in a given area of land devoted to residential development. As discussed in the literature review, we took caution in calculating these density measures because the outskirts of urban areas in Korea are often dominated by forestry, agricultural lands, and other non-residential lands. Failure to account for these inhabitable lands could result in the underestimation of urban density .We used the 2020 travel O/D data at the county level on all transport modes provided by the Korea Transport Database (KTDB) to compute the connectivity measures. We used social network analysis (SNA) to calculate these measures. The form of the analysis is usually either an array of the social networks or a structure classified by different social network characteristics . A varieIn light of the COVID-19 pandemic, the analysis has been employed either to demonstrate the infection network of the confirmed patients and its structural characteristics ,50,51 orOur approach for the social network analysis is different from the aforementioned studies. We applied the social network theory to estimate the spatial structure and its functional characteristics, resulting from the movement of people in daily life using transportation O/D data aggregated at the county level. A city is a place where people travel within and move to and from, and it should not be confined by its administrative boundary ,55. Our We used the centrality measures to quantify how cities exist within a network . SpecifiThe degree of centrality is defined as the summation of the number of direct links upon a node, and, using the number of different links a singular node has, the degree of the node’s centrality can be quantified. In other words, degree centrality tells us if a node takes on a key role within the social network and is positioned in a place of importance. Cities with high-degree centrality can be interpreted as having high linkages, both direct and local, with other cities. Closeness centrality measures centrality based on the closeness of other nodes and differs from the degree of centrality in that it is calculated by adding the distances of the nodes that are both directly and indirectly linked. Since the distances between the nodes are considered, it is used to determine the global relationship of the social network as opposed to direct linkages. Betweenness centrality calculates where a node is situated in relation to other nodes, and if the value of centrality is high, the node can be understood as a mediating city between other cities. Finally, eigenvector centrality measures the comparative score of every node within a network based on the principle that connections to a node with high centrality must positively impact the centrality value of the node in question. In other words, a weight is placed on the connected node, which is then considered in calculating the node’s centrality. Cities with high eigenvector centrality have a high probability of being linked to cities with a high degree of connectivity, which in turn, implies that these cities are more likely to come into contact with other contagious viruses. It is also important to note that if a node is linked to a city with a low connectivity value, the eigenvector-centrality value can decrease.Our control variables included physical environment, socioeconomic factors, and other additional indicators. Hamidi et al.  considerhttp://data.go.kr, accessed on 17 September 2020) and processed using ArcGIS version 10.6 , a geographic information system (GIS) software. Per capita growth regional domestic product (GRDP) was derived from the KOSIS database as a measure of neighborhood economic characteristics. Unlike the Italian study [2) was also included to control for differences in the spatial units. In addition, there was a sudden rise in infection associated with a super-spreader event at a particular church in the city of Daegu. As shown in We followed the methods used in previous studies to measure potential determinants of COVID-19 in Korea. Specifically, we included ratio variables for females and youth (20–29 years old) and the older adults (65+ years old) as demographic characteristics obtained from the KOSIS database. For neighborhood resources, measures of medical accessibility, nursing home service, and park accessibility were obtained from the public data portal  regression model and a robust negative binomial regression (NBR) model. The probability of being diagnosed with COVID-19 exhibits the same probability distribution of a coincidental event. In such a case, a Poisson regression model or a robust negative binomial model are more appropriate to use than a standard OLS model . When usFinally, this paper compared four models—two that differ by the type of model (OLS model and the NBR model) used and an additional two that differ in the definition used for the dependent variable . Model A defined the dependent variable as the total number of confirmed positive cases, and the analysis was based on the OLS model. Model B, similar to Model A, used the same dependent variable but switched the analysis methodology to NBR. Model C, on the other hand, defined its dependent variable as the total number of positive cases per 10,000, and the OLS method was used for its analysis. Finally, we used the same dependent variable for Model D as we did for Model C but used the NBR method of analysis. All analyses were performed in Stata .In terms of the overall model fitness, the AIC and BIC values were smaller in the NBR models than the OLS models, indicating that the NBR models perform better than the OLS models. This implies that a random event could occur in the relationship, affecting the probability of infection. Hence, we focused our efforts on interpreting the NBR model results.When it comes to the connectivity measures, degree centrality was most strongly correlated with the number of cases (coefficient = 45.62 for the OLS model and coefficient = 0.44 for the NBR model). However, none of the connectivity measures were statistically significant for the models that use infection rates. Of the density variables, only net population density was strongly associated with the number of cases (coefficient = 29.38 for the OLS model and coefficient = 0.31 for the NBR model). Again, none of the density measures were associated with the infection rates.Among the population characteristics, the number of cases was positively correlated with the proportion of females (coefficient = 21.16) but negatively associated with the proportion of older adults (coefficient = −0.052) in the NBR model. The OLS model as well as the models that used infection rates showed no statistically significant results with any of the population characteristics. All of the variables indicating neighborhood resources were strongly correlated with the number of cases in the NBR model (Model 2).While the availability of a nursing home was positively correlated with the number of cases, the number of doctors per 1000 people and park area per 1000 people were both negatively correlated with the number of cases, indicating some protective effects of neighborhood resources against the COVID-19 outbreak. This finding is similar to the conclusion of Ma et al. , who fouOur results suggest that the total number of confirmed cases was more appropriate than the infection rates in explaining the relationship with urban spatial structure. One possible reason is that using infection rates in the model could have resulted in canceling out the effect of population density. Because our models included density measures that incorporated population counts in the denominator, dividing the number of confirmed cases by population counts could negate the effect of density and other urban-related factors, leading to the classic type II error. This indicates that further caution will be needed to account for possible biases when modeling the spread of infectious diseases .Using the standardized coefficients, we found that the measures of connectivity were more positively related to COVID-19 infection than the measures of density . This result is somewhat consistent with findings from previous research  that sugInterestingly, the eigenvector connectivity measure had a negative effect on the infection in our model. This was because the eigenvector centrality takes into account the level of connectivity of the connected node (city). Thus, even if the node is connected to a lower-networked city, it has a high degree of connectivity. Both closeness centrality and betweenness centrality were found to be statistically insignificant. This might be because the closeness-centrality measure captures the global connectivity of the social network; therefore, it might not fully capture all the variations in the network condition that are more related to the spread of the virus. Similarly, the betweenness centrality measures the degree to which a node is situated in between different cities within the social network. Again, we believe that this measure is irrelevant in explaining the network condition that is more vulnerable to virus dissemination. On the other hand, the degree of centrality which captures direct connection in the network was found to be the best predictor among other connectivity measures.The net population density was positively and significantly associated with the virus infection. This is consistent with the results of previous studies that found a strong association between population density and infectious disease outbreak ,27. InteThere were a few limitations with the study. First, this study did not consider the influence of social-distancing policies implemented in Korea since February 2020. There were several versions of social-distancing policies throughout the study period, from school closure to restriction on social gatherings. We were unable to examine the effect of social-distancing policies because our data were based on a cumulative record of COVID-19 cases, which did not include time information. The individual public health agency by province only releases aggregate data; therefore, we compiled the data manually from individual government websites at one point in time (17 September 2020). Second, there is a slight possibility that the total number of cases in the early phase of the outbreak could be biased due to the availability of testing. However, some of the undercounting of confirmed cases were resolved as the government ramped up testing capacity early on. In fact, a number of international media praised Korea as one of the few countries with the most aggressive COVID-19 test program . Third, This study provides evidence that connectivity seems to matter more than density itself. Contrary to the common belief, cities are not necessarily more vulnerable to the spread of the disease than the rural counterparts. Instead of focusing on restricting movements in all places, having stricter control over movements and activities in highly connected places could be more effective in dampening the spread of COVID-19.We also found that direct and local degree centrality were more strongly associated with the spread of COVID-19 than the global closeness centrality. This suggests that future mitigation strategies may focus on restricting movements between cities that are directly linked. Despite having a lower effect size than the connectivity measure, the net population density was still statistically significant in explaining the dissemination of COVID-19. Therefore, we cannot ignore the fact that density, especially residential density, could still play a role in the spread of the disease.Our study offers new insights into the current debate on the vulnerability of densely populated cities, especially from the perspective of an Asian country. If our results can be confirmed by prospective studies, they could have implications for developing mitigation strategies to contain future outbreaks. Connectivity and residential density may indeed reflect one of a constellation of factors that need to be taken into account for future planning to protect our cities against infectious disease threats in the foreseeable future. Our study provides important insights into how we might plan and build safer, healthier, and more resilient cities in a post pandemic world.


Ultrasound imaging technology has the advantages of noninvasiveness, real-time, low price, and easy operation. It is one of the most used diagnostic tools for early detection and classification of premature ovarian failure. Although the rapid development of computer-aided diagnosis has provided a great help to the ultrasound diagnosis of premature ovarian failure, it still has many limitations and shortcomings, so this paper adopts transfer learning and feature fusion algorithms to improve the identification and prediction efficiency of premature ovarian failure. In this study, the POF group and the control group both adopted a unified scale. From the four aspects of sociological characteristics, past medical history, environmental factors, and living habits, a dedicated person asked and filled out the scale face to face. All patients participating in the experiment underwent ultrasound examinations. In this paper, the bottom-level feature fusion method is used to improve classification performance. The experiment uses 100 epochs. After each epoch training is completed, we used all the data and labels of the target domain to test. All experiments were performed five times, and the result is the average of five experiments. All the results of baseline and direct classification without migration use the average of five experimental results as the result. Migrating the features extracted by the InceptionV3 network has the best performance for predicting premature ovarian failure. Its classification accuracy is as high as 85.13%, and the F1 value is 0.78. The results show that the migration learning and feature fusion algorithms used in this paper can provide reliable predictive analysis and decision support for doctors in the diagnosis of premature ovarian failure. Under the current medical AI boom, hospitals use advanced equipment to produce a large number of medical images. In the context of big data, people gradually realize the hidden value behind these data and find them through data mining methods. The laws behind them can be used to greatly improve the efficiency of doctors' clinical work. In medical images, the data set is constructed by extracting the regions marked by doctors and extracting samples, thereby training the network model, so that the trained network can assist the doctor to help the patient to recover. However, in these images, not all the information is needed for our actual life and production, which requires the screening and segmentation of the image to extract the specific area we want. At the same time, a large number of collected source images cannot be directly used in theoretical research and production practice most of the time. At this time, certain source images need to be processed and transformed into images that we humans can easily accept and use.This paper proposes an ultrasound premature ovarian failure recognition and prediction method based on migration learning and feature fusion, which integrates highly abstract features extracted by deep convolutional neural networks on the basis of the imaging omics system, and finally combines a variety of feature selection algorithms and model classification algorithms. Realizing the differentiation and classification of premature ovarian failure has positive clinical significance in the classification and diagnosis of premature ovarian failure.Transfer learning and feature fusion algorithms have a certain positive effect on the recognition and prediction of premature ovarian failure by ultrasound. Shi believes that, under actual working conditions, the initial failure of rolling bearings is difficult to effectively predict due to lack of evolutionary knowledge, weak failure information, and strong noise interference. He proposed a rolling bearing initial failure prediction model based on transfer learning and DCAE-TCN. First, he uses a deep autoencoder (DAE as the first two hidden layers and CAE as the last hidden layer) to extract fault features from the rolling bearing vibration signal data. Then, he adopted the balanced distribution adaptive (BDA) method to minimize the distribution difference and class spacing between the extracted fault features and construct a common feature set. He uses the advantages of TCN to extract the time-domain features of the original vibration signal in the target domain. Although his method can effectively learn transferable features, its accuracy is insufficient . Kumar sPremature ovarian failure is a disease with a long course, and the course is affected by many factors, and there are individual differences. Among patients with premature ovarian failure of different ages and different social and cultural backgrounds, the clinical manifestations may have their own focuses. Exploring the clinical characteristics of premature ovarian failure is not only conducive to the standardization of premature ovarian failure treatment but also provides a basis for clinical practice or scientific research. The transfer learning mechanism is applied to the ESR problem, and a new solution is proposed to avoid the cost of retraining the recognition of new scene data and is combined with the SDA network in the transfer learning mechanism to further improve the ESR system based on deep learning. It will improve the system performance. Under the mechanism of transfer learning, a method based on general pretraining is proposed, and transfer learning is applied to the pretraining and fine-tuning of the DBN network. It tries to transfer the knowledge learned from the pretraining to the fine-tuning process. It is innovative and recognizes. The performance is good.n1 and n2 represent the number of samples in the source field and the target field , the calculation of the PFD fractal Fp is as follows:L is the length of the timing signal, and ζ is the number of positive and negative sign changes after taking the derivation of the EEG signal , the calculation of the spectral entropy of the time series signal is as follows:Given the time series data [ density .M, the basis Xt of the source domain subspace can be aligned with the basis Xs of the target domain subspace. Here, the conversion matrix M is obtained by minimizing the Bregman divergence [F represents the Frobenius norm. Because Xs and Xt are composed of eigenvectors corresponding to the first d eigenvalues, they are already regularized, so there is no need to add a regular term in the expression [F norm under the orthogonal transformation, a simple closed-form solution of M can be obtained:Assuming that there is a conversion matrix vergence :(7)FM=Xspression . AccordiM∗=XSTXt can be obtained, and the base of the subspace of the source domain aligned to the subspace of the target domain can be expressed as Xa=XsXSTXt. When comparing samples, the source domain sample γs and the target domain sample γt can be mapped to the corresponding subspace respectively, and then, the subspace can be aligned by the transformation matrix M∗, and the similarity function can be defined [A=XSXSTXtXtT represents the importance of each part of the feature vector in the original space. Sim compares the similarity between the source domain sample γs and the target domain sample γt on the aligned subspace, so the K-nearest neighbor algorithm can be used directly for classification [γs from Xa to the aligned subspace and map the target domain sample γt to its corresponding subspace through Xt and then use the SVM algorithm for classification [Therefore,  defined , 14:(10)fication . In addification .In order to take advantage of the powerful analysis capabilities of convolutional neural networks for pictures and at the same time to overcome its shortcomings of easy overfitting on data with a small amount of data, this paper combines transfer learning and convolutional neural network methods, using methods based on the transfer learning model of convolutional neural network . The conFeature-level fusion is often in the middle of the system, which means that this type of fusion does not limit the matching degree of the sensor. In addition, the data after feature extraction has also obtained effective dimensionality reduction, and the amount of fusion calculation at this time is also reduced. Feature-level fusion can be flexibly inserted into each intermediate structure of feature extraction. In actual multimode systems, feature-level fusion is the most commonly used method of data fusion. However, in the feature fusion, due to the abstraction of the features, the acquisition and selection of the prior probabilities of each channel in the fusion process becomes difficult to determine. In addition to manually setting the weights based on prior knowledge, you can also channel proportion for learning .Before deep learning is widely used, the traditional method of extracting features usually starts from three aspects, namely, color-based feature extraction, texture-based feature extraction, and Yuezi shape feature extraction. However, the previously mentioned three methods of extracting features have their limitations. They often lose sight of the other. Color-based feature extraction focuses on color but lacks spatial information. Texture-based feature extraction only extracts local structured features without color correlation. Therefore, the commonly used method is to merge the previously mentioned characteristics and then analyze them . Data fePremature ovarian failure is shown in A total of 100 patients were selected from the gynecological clinics and reproductive assisted pregnancy clinics and women undergoing physical examinations. There were 100 cases in the normal group and 50 cases in the POF group , 26. TheIn this study, the POF group and the control group used a unified scale. From the four aspects of sociological characteristics, past medical history, environmental factors, and living habits, a dedicated person asked and filled out the scale face to face. Peripheral venous blood was taken during 2–4 days of menstrual cramps, random blood was taken for amenorrhea, and the levels of basic endocrine hormones FSH, LH, E2, T, PRL, and TSH were determined by chemiluminescence method. The height and weight were measured on the day of blood sampling, and the informed consent of the research subjects was obtained for the collection of blood samples, and the relevant informed consent was signed.Transvaginal ultrasound examination is performed on the first 3–5 days of the menstrual cycle , and the examination time for amenorrhea is unlimited. The bladder needs to be emptied before being examined. During the examination, the bladder lithotomy position is taken, the vaginal probe is wiped with a sterile tissue soaked in acidified electric potential water, and a condom coated with a coupling agent is put on. Routine vaginal two-dimensional ultrasound examination is performed first to observe the condition of the uterus and endometrium and then focus on the size of the bilateral ovaries, internal echo, blood flow distribution, and the number of follicles. In the process of 3D image acquisition, the patient holds his breath, and the operator holds the probe in a fixed posture.In this paper, the bottom-level feature fusion method is used to improve classification performance. The specific operation is shown in The experiment process is divided into three stages. The first stage is the data preprocessing stage before the experiment, the second stage is the pretraining stage of the autoencoder, and the third stage is the training stage of the domain adaptation model. In the data preprocessing stage, this experiment uses the preprocessing tool FSL to remove the skull and register the magnetic resonance images. Due to the large difference in size between ADNI and OASIS images, this experiment will register the magnetic resonance image data of the two domains to a standard template, and the image size after registration is 91 × 91 × 91. Then, we used ADNI1 as the source domain data set OASIS1 is used as the target domain data set to start training the model. In the pretraining phase, we randomly removed three source domain data and divided the data into training set and test set randomly according to the method of five-fold cross-validation. The training set contains 80% of the samples, that is, 260 training samples, and the test set contains 20% of the samples, that is, 65 test samples. Use the source domain data to train the autoencoder until it converges and then save part of the encoder model and parameters. In the domain adaptation model training phase, we followed the pretrained encoder with a classifier, a gradient reversal layer, and a discriminator. The experiment uses 100 epochs. After each epoch training is completed, we use all the data and labels of the target domain to test. All experiments were performed five times, and the final result is the average of five experiments. All the results of baseline and direct classification without migration use the average of five experimental results as the final result.x ± s. The comparison of measurement data between the two groups used two-sample t test, and the comparison of count data used χ2 test. P < 0.05 difference was statistically significant.SPSS16.0 software was used to analyze the obtained data. The measurement data were expressed as The comparison of the classification results of different classification models is shown in The cross-modal retrieval mAP@Top10(%) of different methods in SVMD is shown in The relevant indicators of the diagnostic accuracy of the three-dimensional ultrasound parameters in the DOR group and the POF group are shown in The influence of LSTM structure on the prediction and recognition rate of premature ovarian failure is shown in This paper uses Pearson's correlation coefficient for 50-dimensional image features to filter out the relevant redundant features whose correlation is higher than 0.9. On this basis, 167 samples were trained. Under the leave-one-out cross-validation, this paper chooses the method of stepwise regression and Lasso feature selection to obtain the optimal feature subset and trains and optimizes it through logistic regression (LR) and support vector machine (SVM) prediction models. The two methods of feature selection and the two prediction models are combined in pairs to form four sets of prediction model combinations. The prediction results of the model are comprehensively evaluated by calculating the AUC value, standard error , 95% confidence interval , F1 value, and sensitivity and characteristic. The results are shown in P > 0.05). After six cycles of treatment, the uterine volume was significantly larger than before treatment and at three months of treatment (P < 0.05), ovarian volume and antral follicle count (AFC) did not change significantly compared with before treatment.The comparison of ultrasound examination results before and after hormone replacement therapy in POF patients is shown in After fusing the features extracted by Gabor wavelet and convolutional neural network, each sample will get 352-dimensional new features, which consumes a lot of time when using LSSVM for classification. In this paper, we will use the KECA method to reduce the dimensionality of the fused features and compare the experimental results to analyze the impact of KECA dimensionality reduction on the classification results and whether it can meet its requirements. The Jaccard value obtained by reducing different dimensions after feature fusion is shown in Aiming at the problems of poor feature extraction of premature ovarian failure and low recognition rate based on ultrasound images, this paper analyzes and studies the prediction and recognition technology of premature ovarian failure based on the research results of computer-aided diagnosis technology and realizes the identification and classification of premature ovarian failure features. We obtained relatively ideal experimental results, used the trained depth model as a feature extractor to extract and select 27 depth features, integrated 30 texture features that predict clinical individualized information and five clinical features that describe pathological grades, increased the diversity of classification features, and trained three types separately. The characteristic SVM classification model and the output result adopts logistic regression for weighted fusion. After the experiment was tested by data, it was verified by external data. The results show that the migration learning and feature fusion algorithms used in this paper can provide reliable predictive analysis and decision support for doctors in the diagnosis of premature ovarian failure.


This paper presents the development of a new microgripper actuated by means of rotary-comb drives equipped with two cooperating fingers arrays. The microsystem presents eight CSFH flexures (Conjugate Surface Flexure Hinge) that allow the designer to assign a prescribed motion to the gripping tips. In fact, the adoption of multiple CSFHs gives rise to the possibility of embedding quite a complex mechanical structure and, therefore, increasing the number of design parameters. For the case under study, a double four-bar linkage in a mirroring configuration was adopted. The presented microgripper has been fabricated by using a hard metal mask on a Silicon-on-Insulator (SOI) wafer, subject to DRIE (Deep Reactive Ion Etching) process, with a vapor releasing final stage. Some prototypes have been obtained and then tested in a lab. Finally, the experimental results have been used in order to assess simulation tools that can be used to minimize the amount of expensive equipment in operational environments. Microgrippers have been extensively studied for the last two decades as promising tools for a large variety of applications ,2,3. A rSome design methods make use of topology optimization  or pertuAfter the design, many different methods can be successfully used to fabricate microgrippers ,14,15,16The capability of microgrippers has been recently proved in operations of versatile grasping  and autoRecently, microgrippers equipped with Conjugate Surface Flexure Hinges (CSFH)  successfThis paper presents the full development of this device from the stages of design to fabrication, dynamic testing, and simulation.The design of a MEMS device follows a symbiotic development between the device concept and the technological processes that will be used for its realization. A deviced based on MEMS Technology is characterized by maximum dimensions spanning from a few millimeters down to a few micrometers for some of its parts. Therefore, the miniaturization involves different behaviors for the material properties used and consequently also in the electro-mechanical behavior of the parts. On the other hand, it is well known that the smaller the size, the more critical technological issues arise. The object of this work has been designed to be a monolithic microgripper capable of obtaining considerable roto-translational displacements of some of its parts through electro-capacitive actuation (comb-drive). The grasping task is characterized by the type of motion of the the jaws while they are approaching the target object. The most simple layout consists of a pair of symmetric rotating jaws. However, this configuration is rather constraining because the approaching motion to the target is predetermined, namely a simple rotation. Considering the realm of plane mechanisms, the moving links that are adjacent to the frame link present a rotation about a fixed axis. This can be a very limiting constraint to the functional design. For this reason, many grippers have been designed as a compliant mechanism whose pseudo-rigid body equivalent mechanism (PRBM)  is a 4-b40 3–4 handle layer with a thickness of 400 The design requirements suggested the use of monocrystalline silicon as a structural material since it offers mechanical and electrical properties with very good performance. Moreover, monocrystalline silicon shows, at the millimeter scale, a higher elasticity and a lower fragility than that which it presents at the macroscopic scale. The choice of monocrystalline silicon also identifies the type of substrate that will be used: silicon wafer. Furthermore, the need of a material with low electrical resistance and, at the same time, a high mechanical stiffness has oriented the choice towards the SOI (Silicon-On-Insulator) wafer. An SOI wafer with the following characteristics has been the preferred choice to comply with the design requirements:The ideal process for the “bulk micromachining” necessary to obtain this geometric shape is Deep Reactive Ion Etching (DRIE) because it uses an etching that gives rise to a high ratio of the etching gap to the possible depth. In fact, this application needs at least a 40 The device structure is characterized by elements with very different dimensions, and, among these, some elements represent a constraint to functionality. In fact, from the point of view of capacitive electrostatic actuation, it is of fundamental importance that the comb-drive fingers and gaps between the stator and rotor finger arrays have dimensions of a few micrometers see a.From the kinematic point of view, it is necessary that the curved beams and the gap in the CSFH have also dimensions of a few micrometers, as depicted in As the selected technological process (DRIE) is expected to yield a device with both moving and fixed parts, it is of primary importance to manage the silicon etching in order to reach the silicon oxide layer (etching stop layer) in a uniform manner. In fact, the presence of etching areas of different sizes causes the loading effect for the DRIE: with the same process time, the etching is deeper in the large etching opening windows than in the narrow ones.Since the moving parts must be released from the other layers, it is necessary to remove the silicon oxide. However, the underetching, which has limitations due to the micrometric dimensions of the gap, gives rise to the condensation of the etching gas and therefore the collapse of the upper surface on the lower one (stitching effect), stopping the etching process. In order to avoid both of the above-mentioned problems, all the suspended surfaces were modified into reticulated frames, as shown in Quantitative information concerning the adopted comb-drives and CSFH flexures are provided in The microgripper was fabricated on 6” SOI wafers by silicon deep reactive ion etching with aluminum masking. The fabrication process was performed using a previously reported sequence , but theThe fabrication steps can be summarized as follows.The masking layers were initially deposited on both the front and back side in the following sequence: TEOS Silicon Oxide (200 nm thick), magnetron-sputtered Aluminum (150 nm thick), and PECVD Silicon Oxide (300 nm thick). The need for a PECVD layer on top of the aluminum layer was previously described in detail in 2019 .The layers were then patterned with a stepper photolithography process and etched in plasma etching to expose the underlying silicon.The exposed silicon was then etched starting from the front side with deep reactive ion etching (DRIE). Both front and back etchings proceeded down to the buried silicon oxide that is embedded in the SOI wafers and acts as an etch stop.® uEtch, 2021 SPTS Technologies Ltd., Ringland Way, Newport, NP18 2TA, UK).Last, the devices were released by removing the buried silicon oxide with an HF vapor etching tool , which made it possible to measure the angular displacement s of the marker circular trajectory by its radius r, the latter being evaluated as the distance between the marker and the ICR. Then, velocity and acceleration have been computed as the first and second discrete derivatives of the angular displacement, respectively.The processing and analysis of the collected videos have been carried out by an automatic software implemented in a MATLABironment . The disironment . In thisironment ; (d) ideThe main sources of uncertainty have been investigated, as described in some recent investigations ,42,43,44Type A uncertainty, Power supply uncertainty on amplitude, Power amplifier uncertainty on amplitude, Frame time uncertainty, Resolution uncertainty, Based on , it has e 0.4 μm .Software uncertainty, Type B uncertainty, al setup :Power sux and y coordinates that have been processed by a Monte Carlo simulation (MCS) with The implemented in-house software requires that the operator manually selects two pairs of points on the first frame of the video to obtain the ICR coordinates. Therefore, this component of uncertainty has been assessed by asking 12 different observers to perform the described above operation 30 times each, for a total of 360 pairs of combined  to estimConsidering the entire variety of analyzed sinusoidal output periods, the difference between the measured values ® has been used to perform the numerical simulations. Symmetry boundary conditions have been conveniently exploited to reduce the computational costs, so only one-half of the microgripper has been considered  has been adopted to numerically simulate the gripper response when an electric potential is supplied to the electrostatic comb-drives. The commercial software Comsol Multiphysicsnsidered . FEA simthe non-moving fingers are anchored and electrically grounded through the ground pad (A);the moving fingers and the remaining gripper parts (D) can move in the plane with respect to the highlighted pad (B);the edge (C) represents the symmetric boundary condition.The model also considers the air, which surrounds the gripper, as a free deforming domain to properly simulate the electrostatic actuation physics. The considered mechanical and electric boundary conditions are listed below see :the non-As shown in the actual microgripper SOI wafer stiffness matrix is unknown because it is a composite material;The FEA model considers constant out-of-plane thickness and ideal geometry. On the other hand, each device layer shows non-constant thickness, and geometric imperfection is unavoidable in the actual fabricated microgripper. Therefore, the flexural stiffness of the CSFHs (which mostly impacts the simulations) may randomly vary as a result of the fabrication process;the actual device layer presents an aluminum masking that is not considered in FEA.The key parameters in electrostatic actuation simulations are the relative permittivity of the solid and dielectric medium, as well as the elastic modulus of the moving member. In the literature, many works report different values for the Young’s modulus of silicon depending on the doping type and crystal orientation, so the range of values seems very wide . In thiV applied between the fixed (A) and moving elements (D) of the comb-drives  are reported in The values reported in The mean over all periods of the acquired data has been calculated and compared to the value obtained by means of the numerical model. Furthermore, The PE values presented in The above-presented results are quite encouraging for future applications of the proposed method, provided that they are used to calibrate the FEM model. More generally, the method can be extended to other MEMS-Technology-based devices to improve their simulation accuracy and reliability, specially in the field of biological tissues and cells dynamic manipulation, supporting designers to optimize configurations suitable for specific operational environments.The idea of increasing the torque exerted by rotary-comb drives by arranging two cooperative arrays of fingers has been investigated in the present paper. The study was stimulated by the need to develop a microgripper for biological tissue and cell manipulation with a complex structure, consisting of two four-bar linkages in a mirroring configuration, with eight CSFH type flexures. This complexity has been used in the design stage to successfully assign a prescribed motion of the tips. The adopted fabrication process, based on DRIE (Deep Reactive Ion Etching) on a SOI (Silicon-on-Insulator) wafer, was able to yield some prototypes that could be tested in different dynamic experiment. A numerical simulator has been built and calibrated to serve as a tool during future experimental campaigns planned for validation and prototype demonstration in relevant environments. The proposed microgripper design turned out to be promising for low-frequency tissues and cells’ dynamic manipulation.