Advanced driver assistance systems, ADAS, have shown the possibility to anticipate crash accidents and effectively assist road users in critical traffic situations. This is not the case for motorcyclists, in fact ADAS for motorcycles are still barely developed. Our aim was to study a camera-based sensor for the application of preventive safety in tilting vehicles. We identified two road conflict situations for which automotive remote sensors installed in a tilting vehicle are likely to fail in the identification of critical obstacles. Accordingly, we set two experiments conducted in real traffic conditions to test our stereo vision sensor. Our promising results support the application of this type of sensors for advanced motorcycle safety applications. Automotive sensors are key components of advanced active safety technologies for road vehicles. Research has shown that these technologies, named as Advanced Driver Assistance Systems (ADAS), are beneficial for the society as they contribute to mitigate, or even avoid, road crashes ,2,3,4. OThe quality of the remote sensing approaches allowing to artificially perceive the traffic scene for cars  is the cornerstone of current ADAS. Regrettably, these sensors that are common in cars are not implemented in the market of tilting vehicles, not even in the high-end segment. And yet, the safety systems that such sensors enable could have an important impact, due to the high number of tilting vehicles in the fleets at global level ,8,9. CurThe reason that causes the technological gap in tilting vehicles is that current automotive remote sensing sensors fail when they lean over . This liTo address these problems, we conceived a remote sensor designed for motorcycle safety system application. Our design is based on multifocal stereo vision to cover different regions of interest simultaneously, and to guarantee adequate depth accuracy to be part of future ADAS for motorcycles. The importance of this approach is to allow the use of artificial vision methods developed for ADAS in motorcycle safety. Consequently, we aim to provide the way to capitalize on relevant cutting-edge algorithms created for ADAS during the last 20 years and speed-up the development of needed motorcycle safety systems (or Advanced Riding Assistant Systems–ARAS) to make motorcycles and mopeds a safer means of transport. In particular, these stereo vision algorithms for vehicular application are successful in the following tasks: separate objects and surface structures by fast features extraction ; estimatThe remainder of the paper is structured as follows: In addition, four appendixes to support and complement the core ideas of the publication. A proper operation of stereo camera-based sensors requires the use of two synchronized cameras as a whole, which is achieved through the jointly characterization of them . An invariant calibration of the 3D sensor would assume constant parameters of the imagining system, such as relative distance and orientation between the two cameras, which are physically determined by the location of the cameras in the stereo rig.Regrettably, the assumption is not valid for the motorcycle application due to the deformation (micro-bending) of the stereo rig. A motorcycle, as a lightweight vehicle, has less potential than a car to damp the vibrations generated from road irregularities. In fact, in normal riding conditions the vehicle frame is subjected to intense shocks, which passes to the stereo camera rig, producing dynamical changes in the instantaneous distance and orientation between the cameras. As a consequence, an invariant stereo calibration is not suitable for a moving motorcycle setup. In addition, the optical zoom of the long range cameras makes common mechanical anti-vibration solutions ineffective. However, one possible solution is online stereo re-calibration.Concepts of stereo vision considered relevant for the design and implementation of our multifocal stereo camera sensor are recapitulated hereafter. More details on stereo vision and 3D geometrical modeling principles can be found in ,44.Z, see Equation (1)).Estimating depth from stereo imaging is a triangulation task. In the human vision, to solve this task visual information derived from two eyes is used to estimate depth from the so called binocular disparities . The disIf we consider the stereo triangulation in epipolar geometry, the correspondence of points between two images is obtained by means of imaginary scan lines. The distances in pixels along scan lines are the disparities between couples of correspondent points. Epipolar geometry defines epipoles and epipolar lines. For each camera and each point in space, the epipole is defined as the intersection of the camera imager and the line passing from that given point and the focal point of the camera. The line in the space connecting two corresponding epipoles of the two cameras is the epipolar line. Finally, epipoles and epipolar lines have a representation in a rectified space computed using the fundamental matrix ,47, thusEpipolar lines in rectified images are horizontally aligned. This characteristic simplifies the search for matching correspondent features to a simple search within image rows between the pair of rectified images . For reaf expressed pixels unit. The distance between the cameras is their baseline b in distance unit. The difference of the relative projection of a world-point is the disparity d, generally expressed in terms of pixel units. Resulting depth can be computed using Equation (1), which shows that the disparity is inversely proportional to the distance Z (expressed in distance units) of the object.In our setup, both cameras are assumed to have the same focal length FoV) of a single camera is a solid angle through which the imaging sensor is sensitive to the light. Therefore, the FoV define the periphery of the 3D volume of the inspection captured on the camera imager sensor. The FoV depends on a combination between the size of the imager and the camera lens. Therefore, the focal length defines the FoV (Equation (2)), which is related to the focal length f (in distance units) and the horizontal size of the imager h (in distance units).The Field of View . Therefore, in the case of a multifocal strategy it is recommended to use fixed lenses that select a limited Depth of Field  in the desired depth range of measurement, in order to maximize the light sensed from this part of the scene.FoV, as a consequence, it is specified as a Diagonal FoV, Vertical FoV and Horizontal FoV. Additionally, in this paper common FoV is defined as the overlap between the FoVs of the pair of cameras on a stereo rig. The common FoV between the left and right cameras defines the lateral boundaries  of the Range Field (b.f = constant) the possible depth range to perform triangulation is defined for the range of disparities (dmin and dmax), which determines the rear and frontal boundaries of the Range Field.Rectangular imager sensors modifies the concept of circular ge Field . EmployiThe previous top view representation of the Range Field  is a simZ) in a stereo system is defined according to Equation (3), for which Z is the depth in world coordinate frame and ΔZ is the depth error in distance units. The value Δd (disparity step) is directly related to the depth error and this component of the error depends on the capabilities of the stereo matching algorithm for achieve sub-pixel refinement.The triangulation error  inferior to one will decrease the triangulation error, or for the same error the triangulation Range Field can be extended. On the other hand, a fractional value of disparity can be seen as virtual expansion of the baseline (b). In this second regard of fractional Δd, for example, a portable stereo sensor designed to assist the visually impaired has achieved 1/8 sub-pixel resolution in an embedded system, compensating for the short baseline allowed for the wearable application [From Equation (3), it is clear that a disparity value  to set an equivalent value of 1/2 of sub-pixel accuracy with respect to the imagers in the aforementioned literature. This conservative decision limits the full range of the measurement at expenses of a gain in robustness against sub-pixel camera decalibration .Depth triangulation in stereo camera systems depends critically on accurate calibration of each camera pair and in a vibration-free set up. The calibration is constituted by intrinsic and extrinsic parameters, for which the latest refers to the relative camera pose (3D orientation) between the two cameras. The extrinsic calibration depends on the physical fixation of the pair of cameras during time. In this regard, a motorcycle is a harsh environment where vibrations coming from the engine (which is rigidly fixed to the motorcycle frame), the vehicle-road interaction, and aerodynamic drag forces can slightly modify the instantaneous pose between the camera pair along time (that cannot be seen with the naked eye).Depending on the application, a variety of techniques are used to solve this issue. Examples: (a) In visual odometry it is usual to perform continuous stereo extrinsic re-calibration  operating on sparse stereo correspondences on stereo frame basis ; (b) In The results corresponding to the Disparity Map (DM) calculation may have associate a level of confidence meaning that, for each pixel of the DM a probability that express how real is the triangulation can be associated. Non-real triangulations due to lighting reflections or circumstances of bad visibility, like rainy weather, can lead to wrong detections. Several metrics were developed as a way of quantifying the stereo confidence; a first framework for stereo confidence clue evaluation defined a taxonomy  that wasThe stereo confidences is a valid measure that can be used in absence of ground truth data . This isThe variety of stereo confidence metrics perform differently in varying outdoor conditions, thus the wise fusion of them implies more robustness against stereo matching errors. Machine-learning approaches allow to use a set of metrics to improve the accuracy of the stereo confidences ,87,88,89We provide a detailed explanation of the design considerations of our camera-based remote sensors and consequently, how the system was evaluated for the application in motorcycle safety.The multi-focal stereo rig is shown in The synchronization between the six cameras (Camkong) of the lower rig was performed by hardware , while tThe design of the remote sensor initially requires to define the spatial zones at the front of the scooter which are necessary to scan, as it is shown in Applications in advanced safety systems like the conceptual Motorcycle Autonomous Emergency Braking (M-AEB), requires a precise triggering to avoid false positives. In particular, for M-AEB safety system, the depth resolution required for the proper identification of the inevitable collision state was defined in a spatial grid of 20 cm . Thus, oFoV) of the lenses focalize the scanning volume in a narrow frustum.The short-range stereo pair have fisheye lenses to scan a wide 3D space ahead of the vehicle. The light arrives to the imager from multiple directions, these motivated to define a short Range Field for the stereo triangulation. The case of the long-range cameras differ, because the narrow Field of View  enclosing the desired frustum, as shown in FoV and the focal length was chosen  of all stereo pairs that were recording video concurrently. Consequently, the analysis of the footages of each camera allowed to picking up the suitable stereo frames (these that contain a complete view of the checkerboard in the couple of images that conform the stereo frame) to perform the stereo calibration for each pair of cameras.Essentially, we moved the checkerboard specified in  throughoNext, we started the corner detection process, searching for symmetrical corner features in the images in order to find the checkerboard. State-of-the-art subpixel accuracy algorithms  contribuIn On the top of Finally, on the bottom right of The main camera parameters obtained as a result of the static camera calibration performed for both stereo sensors are shown in FoVs, baselines, resolution of the imagers, proportioned a baseline with the ideal measurement capabilities of the stereo vision sensors. However, several practical factors can affect the measurement range of a stereo camera system and this experiment was designed to quantify the Range Field of all the stereo camera rigs.The design considerations of the imagining system developed, mainly in terms of focal lenses, common The test was conducted in an open and flat surface after having delimited a rectilinear corridor by traffic cones a. The coThe 3D reconstruction of the scene imaged c shows tThe point cloud calculations are not supposed for a real time application, it is only used for helping to assess the quality of the 3D information measured. For the top view of the scene d the poiThe bias error of depth grows for longer distances as is expected for Equation (2), this is being negligible for the first two pairs of cones and a tolerable 1.68% and 2.25% systematic errors  application  can be aInvariance of intrinsic parameters of each camera.The extrinsic parameters of each camera pair varies within a small range.The vibrations in the scooter temporally misalign the cameras of the calibrated stereo camera sensor . Therefore, an online re-calibration is needed to perform triangulation. The re-calibration implemented is based on a two-step rectification under two assumptions:In this way, the images are rectified two times to avoid problems of scaling. The first time uses the camera parameters obtained in the static calibration , and a sNevertheless, as multiple correspondence measures are available and matching methods can lead to significant differences in matching results, there is a trade-off between execution time and descriptor quality to be evaluated empirically ,103,104 As starting step, we decided to perform a qualitative comparison by employing two different kind of keypoint descriptors, a histogram-based descriptor and a binary descriptor. It is worth saying that the two feature descriptors chosen have shown good performance in real-time implementations (comparison in The election of SURF (Speeded-Up Robust Features) method ,107 as iIn Up to now, we implement the online camera re-calibration employing SURF features and SURF feature descriptors. However, binary descriptors are more suitable for real-time applications because they requires significantly less memory than histogram-based descriptors like SURF. This motivated our second re-calibration test employing SURF features and FREAK (Fast REtinA Keypoint) descriptors. FREAK is based in human retina behavior .A real example of dynamic decalibration acquired from our stereo camera rig is presented in We designed two experiments to assess the capability of our custom built remote sensors to perform depth and orientation measurements of target cars in a real traffic scene. All test were conducted in outdoor and clear visibility conditions, being the most common circumstance of motorcycle crashes in Italy . TheThe first experiment consisted in a turning maneuver. It is known that automotive remote sensors are not designed to operate when the instrumented vehicle is tilted and in such conditions they would fail . This exThe second experiment analyzed a pre-crash scenario (pre-crash test) and consisted in emulating the trajectories of a scooter and a car that led to a documented crash ,109. TheThe details of the test protocol and the instrumentation of the vehicles were described in a previous paper . In addiThe following results for each of the two experiments are presented in the form of Disparity Map (DM) and 3D point cloud (3D reconstruction).The following results  correspoBoth DMs depicted in the figure show the accurate detection of the ground, which is used to define a ground plane and from that, the tilted Region of Interest (RoI) for the subsequent analysis. In addition, the different range values contained in the DMs are consistent with the type of lenses and baselines employed for the short- and long-range sensing, respectively.In the short-range sensor (the shorter baseline of the stereo camera rig), the fisheye lenses used embraced the light coming from a wide volume of the 3D space into the imaging sensors. Consequently, the maximum value of the disparity becomes big see . In FiguFoV) and longer baseline of the stereo camera rig produced a smaller range of values of the disparities .Note: the 3D point clouds presented until now contain all the raw data measurements retrieved from the stereo cameras. However, certain details are difficult to assess the 3D measurements. Therefore, in order to simplify the assessment of the quality of the measures corresponding to the long-range sensor, we will “clean up” the 3D point cloud. The cleaning process consisted in two steps:In this manner, we extracted and inspected a RoI above the ground plane as it can be seen in In The results presented in FoV and the generous horizontal and vertical spatial discrimination offered by camera-based sensors. These two key design specifications allowed us to measure the relevant depth information of the traffic scene while the remote sensor was subjected to the dynamics of a tilting vehicle (tilted 13 degrees), as was never reported before.The proposed remote sensing strategy was designed to handle changes of orientation (tilting) of the sensor by exploiting the wide diagonal In a previous publication  was defiFor the present study we emulated one particular real-world motorcycle crash and tested the capabilities of our remote sensor based on stereo cameras to sense the conditions of the traffic scene in the neighborhood of the point where the collision became inevitable. In particular, the experiment consisted in remotely measuring the distance an orientation of the opponent vehicle (a passenger car) from the sensors in the scooter. For visual help, the reference point of deployment of M-AEB was marked on the road surface with a white “x” as visible in In this experiment (pre-crash test) depicted in In the 3D anaglyph of From the results shown c,d the sA clean 3D point cloud  is presented in It can be seen that part of the supporting frame of the satellite marker appears in the reconstruction. These results show that the remote sensing strategy is able to measure the car pose and narrow obstacles in real conditions even in the presence of roll angle fluctuations , that alThe temporal charts  presentsFor the reproduction of our results and for algorithm benchmarking purposes, we provide an online image dataset with ground truth  belonginAs explained in The The continuous and dashed lines of the chart are used to distinguish between the totality of the pixels used to compute the DM and the most relevant set of disparities calculated, which are above of the Horopter 10d. For the two online re-calibration cases, the percentage of pixels used to calculate the DM appear to be stable around 70% to 74%, which is interesting and convenient. Also, in these cases the number of disparities below the Horopter 10d are negligible and consequently, the continuous and dashed lines in the chart are almost overlapping.The overall goal of our research activity was to contribute in making motorcycling safer by fostering the implementation of assistance technology. This technology can be transferred along to a variety of powered vehicles presenting a tilting motion, such as scooters, mopeds, mofas, and three- or four-wheeled tilting vehicles denominated Narrow Track Tilting Vehicles (NTTVs). This is important because the injury severity level in case of crashes is the biggest barrier for the societal adoption of tilting vehicles. In the case of motorcycle crashes, these types of road users are generally subjected to serious consequences for life. Motorcyclist are the 23% of the deaths on the world’s roads , and they have 26-fold higher risk of death than those driving other types of vehicles .FoV (Field of View) and additional desirable features, such as resolution, lightweight, passive (no increase to electromagnetic pollution), low consumption and affordable cost.Consequently, motorcycle safety systems to support people to avoid crashes must be enabled. Previous research identified a technological limitation on automotive remote sensors, for which these sensors cannot operate on a tilting vehicle . To overIn the present study, we targeted the development of a novel multi-focal stereo camera sensor to provide remote sensing able to operate under the constraints imposed by the motorcycle dynamics. The importance of the technological sensing solution we proposed relies on the potential to bridge the technological gap that causes the existing lack of rider assistance technologies for tilting vehicles. An example of application of sensing technologies for improved safety of motorcycles was provided with the emulation of a real motorcycle crash, as described here and in . These tIn a camera-based perception system, the quality of the camera sensor is essential. The proliferation of mobile phones with camera sensors during the last decade, reinvents the way in which camera sensors work, as well as their performances. A proof of that is the ongoing standardization of the camera sensors for the automotive industry , which started to work over the advanced draft of the current IEEE P1858 Standard for Camera Phone Image Quality. For related information, an overview of the image quality test for phone cameras is presented in .Another important consideration is that the image sensor manufacturing technology is well below its limits. Image sensors use CMOS technologies that are at least 2 generations behind those of solid-state memories or digital integrated circuits. Thus, in the next few years several innovations in camera sensors are likely to take place. In particular, HDR imagers (High Dynamic Range camera sensors) are showing impressive capabilities using inexpensive technology . Recent On the stereo algorithms side, the algorithms can calculate disparities in regions where there are no specularities or occlusions. In regions with low contrast or with high sensor noise, most implementations have difficulties. However, specific implementations allow 3D perception also in adverse weather like rain ,114,115.In addition, stereo cameras allow the discrimination of water on the road (possible slippery surfaces) by polarization light filters installed in each camera and machine-learning methods ,117. AnoIn our experiments, the performance of the sensor developed employing low-cost action cameras was satisfactory in a static setup. The sensor showed good potential for the application in advanced motorcycle safety systems as it was able to measure small targets sized 30 cm of height (traffic cones) from a distance up to 21 m and road curbs during the test in the traffic scenarios. This sensing capabilities are promising for motorcycle safety application, for which unexpected small obstacles in the travelling path or occasional slippery surfaces can cause serious consequences to the motorcyclist .In this paper we mentioned several promising state-of-the-art solutions that are include machine-learning techniques. These technics require an intensive computational power that consumes significant amount of energy that will not be available in a tilting vehicles. Fortunately, recent embedded neural computers designed in a single chip (ASIC: Application-Specific Integrated Circuit), provide the capability to deploy certain Deep Neural Networks (DNNs) which a power consumption inferior to 1.2 watts. Other relevant aspect is that these ASICs have a Vision Processing Unit (VPU) to process camera data in real-time. This is important because all the algorithms used in this paper can be implemented in VPUs. These new chips and the upcoming improvements in camera sensors are promising technologies to make remote sensors camera-based, such as the one developed in this paper for motorcycle safety application.The use of stereoscopic vision for motorcycle safety as the single remote sensor is possible, assuming clear visibility conditions  in most urban crashes. The important aspect of this paper relies on the whole explanation of an affordable, lightweight, and low power consumption remote sensor able to 3D measure the environment from a tilting vehicle (the scooter demonstrator). Particular attention was also given to the operation of the sensor under fluctuations of the roll angle because, these fluctuations are present in the motorcycle dynamics, even when the vehicle is traveling straight and on flat asphalt (up to 4 degrees in the pre-crash scenario).Additionally, the operation of the camera-based sensor was analyzed when the scooter cornering (up to 13 degrees in the turning maneuver). Rotation between reference coordinate frames, belonging to the host vehicle and the road obstacles, means that existing automotive sensors solutions do not work in the tilting vehicle domain .This publication is the first integral approach to implement a remote sensor capable of enabling ADAS specifically for motorcycles with the potential to avoid crashes, the so-called ARAS: Advanced Rider Assistance Systems. This is an important step towards the concretization of preventive safety technologies in tilting vehicles. Vehicles that cannot offer important crash injury mitigation levels as passenger cars.The feasibility study for the triggering of a motorcycle Autonomous Braking System (M-AEB) from our camera-based sensors started in  and contThese results also seem prominent for the application of pose accuracy estimation methods that have the potential to accurate estimate the heading angle of the opponent vehicle. To this end, we provide material to the research community in the form of an image dataset for the 1 second pre-crash test , with thDuring the experiments in the real traffic, we notice that the roughness of the road can negatively affect camera-based sensors performance. In particular, stereo camera sensors are more susceptible than monocular ones due to the extrinsic decalibration phenomena, which makes depth measurement unusable. As a consequence, we strongly suggest that future assessment of camera-based ARAS and ADAS, such as those of NCAP (New Car Assessment Program), should be carried out on different types of cobbled roads instead of just on the asphalt.Finally, the satisfactory results achieved so far and presented in this paper warrant the validation of our remote sensor in a variety of different conditions to become a part of a safety system. For example: assessing the sensor and its algorithms under different levels of vibrations, the sensor response in tilting vehicles of different characteristics, the utilization of different kinds of imagers , and the assessment of the remote sensor in a wider range of motorcycle crash scenarios.


In the paper, we propose a novel object-oriented hierarchy radiation consistency method for dense matching of different temporal and different sensor data in the 3D reconstruction. For different temporal images, our illumination consistency method is proposed to solve both the illumination uniformity for a single image and the relative illumination normalization for image pairs. Especially in the relative illumination normalization step, singular value equalization and linear relationship of the invariant pixels is combined used for the initial global illumination normalization and the object-oriented refined illumination normalization in detail, respectively. For different sensor images, we propose the union group sparse method, which is based on improving the original group sparse model. The different sensor images are set to a similar smoothness level by the same threshold of singular value from the union group matrix. Our method comprehensively considered the influence factors on the dense matching of the different temporal and different sensor stereoscopic image pairs to simultaneously improve the illumination consistency and the smoothness consistency. The radiation consistency experimental results verify the effectiveness and superiority of the proposed method by comparing two other methods. Moreover, in the dense matching experiment of the mixed stereoscopic image pairs, our method has more advantages for objects in the urban area. With the development of satellite imaging for Earth observation, it is feasible to obtain high-resolution multi-angle, multi-temporal, and multi-sensor satellite data. A highly precise 3D reconstruction by making full use of the multi-source data is one of the important research hotspots in the remote sensing field . EspeciaFor the mixed stereoscopic image pairs of different temporal, there is no specific radiation consistency method to solve the problem of different illumination for dense matching. However, several methods have been developed for the radiation consistency of multi-temporal remote sensing images in other applications, such as change detection and image mosaicking ,3. In geFor the traditional stereoscopic image pairs of the same sensor, several literatures pointed out that noise has an important influence on dense matching. Crespi et al. used CARTOSAT-1 satellite stereoscopic image pairs to analyze the influence of noise distribution on dense matching in 3D reconstruction and proposed that image denoising is helpful ,12. PateThe purpose of our paper is to improve dense matching for different time and different sensor stereoscopic image pairs. Moreover, dense matching is a crucial step in 3D reconstruction. In other words, better results of dense matching can result in better 3D reconstruction. In this paper, there are three main contributions. Firstly, we have comprehensively analyzed various influencing factors of different temporal and different sensor images for dense matching. Additionally, the hierarchy radiation consistency method is proposed to solve illumination consistency for different temporal images and smoothness consistency for different sensor images. Secondly, the object-oriented algorithm idea is proposed to obtain the radiation consistency in more detail, which is especially helpful for dense matching of urban areas. Finally, the smoothness consistency method using union group sparse is first proposed to obtain similar smoothness levels between different sensor stereoscopic image pairs. Therefore, our hierarchy radiation consistency method is beneficial for 3D reconstruction.The remainder of this paper is organized as follows. The purpose of the proposed object-oriented hierarchy radiation consistency method is to obtain radiation consistency of different time and different sensor images for improving area-based dense matching. As shown in For smoothness consistency, we propose union group matrix to simultaneously control the smoothness levels to be similar for the mixed stereoscopic image pairs. For illumination consistency, we propose a two-layer method to solve illumination uniformity for a single image and illumination normalization for image pairs, respectively. In the illumination uniformity step, the relative gradient method is employed to improve the radiation quality of single image. This is the second level in the blue boxes of For illumination normalization, coarse-to-fine relative normalization method is proposed. In the coarse step, the improved singular value equalization is employed for the initial global normalization. In the fine step, an object-oriented illumination normalization method based on piecewise linear correction is proposed to adjust the details for each object. This is the third level in green boxes of In addition, object region association is the premise of the whole object-oriented method. For object region association, the remote sensing image segmentation method based on occlusion random texture model (ORT) is employed to extract the object areas of each image . Then, sFor high-precision 3D reconstruction, the accuracy of dense matching in object areas is extremely critical factor. In the urban areas, it is difficult to obtain successful matching due to the uncontinuous elevation information of building objects. Moreover, the mixed stereoscopic image pairs of different time and different sensors have different radiation properties, which are almost impossible to dense match, especially in the object areas. Therefore, the paper proposed a novel grayscale consistency method, which not only considered the whole image but also mainly focused on object-oriented uniformity. It is very helpful for the accuracy of dense matching in particular with object areas.Object extraction is a significant part in our object-oriented method. McCann et al.  proposedThe multi-temporal data is obtained by different illumination conditions and is the most characteristic. Moreover, different illuminations must lead to the difference in grayscale characteristics between the multi-temporal stereoscopic image pairs. It will increase the difficulty of stereo matching, which will have a significant influence on the accuracy of 3D reconstruction, especially in the object areas. In the paper, we proposed to adjust the illumination difference in two ways. On the one hand, the relative gradient method is employed to obtain illumination uniformity for a single image. On the other hand, illumination normalization method from global to objects is proposed to eliminate the grayscale difference between the multi-temporal mixed stereoscopic image pairs.µ is a constant, it is set to 0.1, and The purpose of this paper is to improve the radiation quality of remote sensing image for dense matching and 3D reconstruction. In fact, different illumination is only one of the factors influencing different radiation characteristics of multi-temporal images. The overexposure or overshadow of the image local areas also has a great impact on the matching. Therefore, the relative gradient is used to improve the contrast and texture details of the object areas for illumination uniformity images, which are very helpful for dense matching. The original gradient of the image is adjusted to make the larger gradient suppressed and the smaller gradient stretched. The adjusted gradient can be expressed asIllumination normalization to eliminate the difference of grayscale characteristics between the multi-temporal stereoscopic image pairs is an important part in the paper. This part is equal to relative radiation normalization in many types of research, which only focus on relative grayscale consistency of multi-temporal data. In early studies, many researchers used histogram matching and gamma collection method to obtain similar gray statistics of images. In recent years, for change detection and images mosaic, many people proposed that objects in the same area of different temporal images constitute spatial homogeneity, which is assumed to be the linear relationship. Based on these studies, this paper proposed two stages of illumination normalization algorithm from global to objects. Firstly, the global illumination consistency is corrected based on grayscale statistics information. Secondly, the gray values of object areas are further refined by the linear transformation relationship using object region association results.SVD  [D, V, and I. D and V are both Orthogonal Matrix. The column of D and V are left-singular vectors and right-singular vectors of I, respectively. A is the grayscale mean of multi-temporal mixed stereoscopic images. The new image A is singular value decomposed. Using the max singular value A, and The improved singular value equalization is employed to obtain the global illumination normalization results. The singular value equalization is used based on on, SVD) . For anym represents the number of object areas, k is the position of the pixel, y is the gray value of the reference image, and x is the gray value of different temporal image in the corresponding position. In the field of study on the relative radiation-normalization of different temporal data, many researchers like Zhang  and ZhonN is the number of matching points in every matching object area, and Because each matching object area has multiple matching feature points to calculate the linear coefficients, kth pixel in mth matching object area, and mth matching object area. The value of the highest frequency disparity (l is the threshold. It is 2 in the paper.From the formulation (11), Satellite panchromatic images are almost obtained by TDI-CCD. The different satellite sensors used TDI-CCD with different storage units designed by different companies, which can lead to the different smoothness levels of images. Almost any reference shave mentioned that the noise levels affect dense matching results. The purpose of smoothness consistency in the paper is to achieve similar noise level for different sensor stereo image pairs. We propose a method based on simultaneously controlling two image noises on group sparse representation for the different sensor stereoscopic image pairs. The sparse model performed excellently in the image restoration, which can suppress noise and protect image structure features. However, in the traditional sparse representation, the patches are independent and cannot build a relationship between similar patches belonging to the same objects ,23. Moret best matched patches in the nonlocal space of the single image. I, which means n is the number of groups, and it is obvious that each patch corresponds to a group. Like traditional sparse algorithm, group sparse model can be formulated into the following minimization problem [p = 1, q = 2. The group sparsity regularizer ith row of matrix m = t + 1 denote t matched patches D for the entire image in the traditional sparse algorithm.For each patch  problem ,26:(14)oblem as , rather τ. The same threshold τ and the similar diagonal matrices can make two different sensor reconstructed images keep the similar information, which can obtain the same smoothness level images as much as possible. Many researchers refer to the effect of noise on dense matching. Moreover, the mixed stereoscopic image pairs are obtained by different TDI-CCD sensors with different designs  leading to different smoothness levels, which have more of an effect on dense matching. We propose union group matrix to simultaneously suppress the noise of different sensor stereoscopic image pairs for the same smoothness level, which makes use of extracting similar patches in the group sparse model to build the relationship between two different sensor images. The proposed method is as shown in In order to construct a union group matrix, the corresponding group matrices of two different sensor images with similar patches are obtained at the same time. Assuming that, the two different sensor images are s1 and s2. Any patch of the reference image s1 in the matching object areas are considered as central patch. The rules that construct the union matrix mainly include the following two points. (a) The central patches of s2 are obtained on the corresponding position with the distance m is the number of columns in the union group matrix, and Y is the original union image. k + 1) iteration. The τ is used to threshold the singular value matrix of two original group matrices, which can control the smoothness level of two different sensor images at the same time. The final denoised images are obtained based on iterate regularized, which makes the filtered noise of each iteration feedback to the image. It can be expressed asy is the original image, and k + 1 iteration result.The group sparse model as standard low-rank approximation problem is solved by SVD, whose core problem is to determine the threshold he paper , for theTo evaluate the performance of the proposed method, we conducted three groups of experiments on the mixed stereoscopic image pairs of different time and different sensor. The first and second groups of data are both WorldView-2 and QuickBird images. The mixed stereoscopic image pairs are from different time and different sensors. The study scene is different areas over San Francisco in the USA. The WorldView-2 images were acquired on the 9 October 2011 with 0.5 m resolution, and the QuickBird images were the 11 November 2007 with 0.6 m resolution. In our experiments, the images were resampled to 0.5 m resolution. The last data is different temporal BJ-2 satellite images with 1 m resolution over Beijing in China. The images were taken on different months in 2016 from 21st-century space Technology Application Co.,s Ltd. . All the experimental stereoscopic image pairs have different illumination properties and some have different noise levels. The mixed stereoscopic image pairs are processed to obtain radiation consistency for dense matching, which demonstrates the efficiency and robustness of the proposed algorithm.j is the value of grayscale, j, M is the size of the image. When the C value is closer to 1, the two object images own the higher the consistency of the gray value distribution.From the subjective analysis above, the two compared methods can adjust the overall illumination characteristics of stereoscopic image pairs to almost the similar level, while our method can recover not only the entire images but also object areas to similar level for illumination consistency and smoothness consistency. For the highly accurate 3D reconstruction, the grayscale consistency of object areas is important for dense matching. This group of experiments used improved histogram similarity for quantitative evaluation of the object radiation consistency. The formula of improved histogram similarity is shown as (20).In order to reduce the influence of background information around the object areas, the minimum external rectangle of every object is cut to be used. Additionally, the background regions are set to 0. From the subjective analysis, the correct matching rates of dense matching experiments are all relatively low. On the one hand, radiation quality between the two stereo image pairs still exists differences. On the other hand, it is the main reason that all the stereoscopic image pairs are different temporal images. Except for larger building objects, the background, roads, and some traffic have substantial changes, which lead to the changes in image content. In fact, the different contents cannot be matched, and the overall matching rates are all very low. Firstly, comparing (1) with (2)–(4) in In order to further demonstrate the advantages of the proposed object-oriented hierarchy radiation consistency algorithm on the dense matching of object areas, the quantitative analysis of correct matching rate in each object area is presented. As shown in A novel hierarchy radiation consistency method is proposed based on the comprehensive analysis of different temporal and different sensor data. For the different temporal stereoscopic image pairs, the illumination uniformity for single image and relative illumination normalization for two images are both considered to obtain the illumination consistency images. For the different sensor stereoscopic image pairs, different smoothness levels are solved by the proposed union group sparse method. Our hierarchy method simultaneously controls the illumination consistency and the smoothness consistency, which can be very helpful for dense matching.The object-oriented method idea is proposed. The object extraction method and feature-based sparse matching method are employed to build a relationship between the same object areas in the mixed stereoscopic image pairs. Additionally, our radiation method can be carried out in the corresponding object areas, which can obtain the radiation consistency images in more detail. The object-oriented method idea is beneficial for the dense matching of building objects in urban areas.In the smoothness consistency step, a union group sparse method is proposed based on the original group sparse model. The two different sensor images are improved to similar smoothness levels by the same threshold of singular value.In this paper, we have put forward a novel hierarchy radiation consistency method for the mixed stereoscopic image pairs of different time and different sensor. Our work more fully considered multiple influence factors of different temporal and different sensor data on dense matching, which can control both the illumination consistency and the smoothness consistency. From the experiment results, our method obtained better consistency than the other two methods, especially in the object areas. Moreover, the dense matching experiment results proved once again that our method has a great advantage at acquiring the better correct matching rate of the mixed stereoscopic image pairs in object areas. The main contributions in the paper are summarized as follows:In conclusion, we proposed the object-oriented hierarchy radiation consistency method first, specifically for dense matching in the 3D reconstruction. The illumination consistency and the smoothness consistency images are obtained using our method, which can improve the application potential of different temporal and different sensor data in the 3D reconstruction field.


Australopithecus africanus fossil, the ‘Taung Child’. The hominin was recovered from a distinctive pink calcrete that contains an abundance of invertebrate ichnofauna belonging to the Coprinisphaera ichnofacies. Here we describe the first fossil bee’s nest, attributed to the ichnogenus Celliforma, from the Plio-Pleistocene of Africa. Petrographic examination of a cell lining revealed the preservation of an intricate organic matrix lined with the calcitic casts of numerous plant trichomes–a nesting behaviour unique to the modern-day carder bees (Anthidiini). The presence of Celliforma considered alongside several other recorded ichnofossils can be indicative of a dry, savannah environment, in agreement with recent work on the palaeoenvironment of Plio-Pleistocene southern Africa. Moreover, the occurrence of ground-nesting bees provides further evidence that the pink calcrete deposits are of pedogenic origin, rather than speleogenic origin as has previously been assumed. This study demonstrates the potential value of insect trace fossils as palaeoenvironmental indicators.The Buxton-Norlim Limeworks southwest of Taung, South Africa, is renowned for the discovery of the first  Australopithecus fossil, the juvenile ‘Taung Child’ at the Buxton-Norlim limeworks ),There is very little ecological or environmental information available for the Taung locality to date, and even less specific to the Type Site. The majority of palaeoenvironmental assumptions following analysis of the ‘Taung Child’ were based on faunal remains and sediment analyses. A number of small- to medium-sized animals have been identified at the site, all of which are well adapted to living in cave, rocky, or edaphic microhabitats . This acThis paper will attempt to identify the inhabitants of a bee’s nest recovered from the PCS of the Type Site at Taung using the literature on classification that is available and comparison with other fossil bees’ nests that have been described in publications. The presence of the ichnospecies and ichnofacies identified at the site will be used in conjunction with other published data to evaluate the likely palaeoecology and palaeoenvironment of Taung Type Site in the Plio-Pleistocene. Finally, any implications that the presence of a species of ground-nesting bee may have on the interpretation of the Type Site deposits will be explored.The majority of modern-day bee species nest in the ground and are known to construct their nests in a diverse range of soil types including (but by no means limited to) sandstone, clay, alluvial silts, desert sand dunes, and beaches . The celBees’ nests can often be found in clusters; this is common due to the fact that solitary bees search for sites with specific qualities . Nests aAlthough dependent upon the species, the majority of these nest makers line their cells with an earthern layer, such as a fine clay; they then smooth the surface with the pygidial plate, and apply a cellophane- or wax-like  substance that is secreted by the bee . Both thCelliforma that, at the time, included all fossil bees’ nests recorded to date [Celliforma proposed by Brown (1935) included “all fossil fillings of chambers purporting to have been made originally by unknown mining Hymenoptera” [Although fossils of bees are rare due to their delicate body structures , the tra to date , 18. Theby Brown 35 includCelliforma, but, in accordance with recent refinements to the classification [Only one other fossilised bees’ nest has been recorded from present day Africa. Thackray 1994) described a nest of sweat bees from Rusinga Island, Western Kenya, which dated back to the Miocene . This ne94 descrification .The identification of insect trace fossils and ichnofacies can prove extremely useful to studies of palaeoecology and palaeoenvironment. Insects often incorporate organic matter in to their nests and, in consequence, are enormously important to the process of soil formation; an insect nest can be a very reliable sign of soil development and thus can provide clear evidence of the presence of paleosols . SensitiThe identification of cell linings in fossilised bee cells can also provide important information with regards to the local environment. Cell linings can provide a defence against unfavourable conditions and might be incorporated to prevent water from entering the cells . Few groAs the microclimate and vegetation are dependent on the climate as a whole, the nests can provide valuable information about the palaeoclimate and palaeoecological conditions . InsectsSouth African Heritage Resources Agency permit number 80/09/10/028/51 was used for the export of geological specimens from Taung. These specimens include: TDTF 1, 2, 2A, 2B, 2C, 3, 4, 5, 5A, 6A, 6B, 6C, and 6D. The specimens belong to the permanent university museum repository at the Fossil and Rock Collection of the University of the Witwatersrand, Johannesburg, Evolutionary Studies Institute, PO Wits, 2050 Wits, South Africa.The fossil bees’ nest described by Thackray (1994) was prepared by sawing 14 parallel cuts through the nest and recording the 3-dimensional pattern of cells and tunnels . Rather The CT scans provided a series of 2D slices that were transformed into a 3D digital model using Drishti software . ExperimThe nest and visible individual cells were measured, and their shapes, orientation and pattern recorded in order to provide an overall description of the nest as a whole. It proved difficult to determine the proximal from distal ends of the cells that were still encased in the matrix. There were also only a small number (25) of visible cells on the dorsal surface of the nest, making it difficult to observe the density and pattern of cell distribution within the calcrete blocks.A visible cell from the underside of block 1 that had been broken in situ was thin sectioned for petrographic analysis. The section was prepared at Birkbeck College’s Department of Earth Sciences, and then viewed with a petrographic microscope under plain polarised light. Photographs were taken to document the structure of the cell lining. The sample was also viewed under a confocal microscope at the UCL Confocal Microscopy Unit, with the use of Leica Lite software in an attempt to see whether or not the trichomes within the lining emitted fluorescence as an organic structure.The whole specimen is approximately 115cm in length and 50cm in height when pieced together see . There aA few cells can be confidently identified from within the sediment blocks using micro-CT e.g. , but theThe individual bee cells are flask-shaped (with the inclusion of the unlined entrance burrow). The main compartment is oval-shaped with very smooth, polished walls and a distinctive calcite lining of approximately 1mm in width see ; the celPetrographic observations indicate that the organic cell lining has been replaced by calcite, but that relics of the organic lining have been preserved. Calcite replacement has occurred at an early stage of diagenesis, as indicated by the preservation of both the organic lining and calcitic casts of plant hairs (trichomes). There is no evidence for in-filling of the cells by a drusy calcite cement, indicating that the cell linings are of a similar thickness and morphology to that constructed by the bee.Inside the cell lining are a large number (around 55) of distinctive hair-like structures see Figs  and 7. TThe smooth, polished walls of the cells and the organic features of the lining allow for the attribution of the nest to a bee, rather than a wasp . The celPalmiraichnus and Celliforma [Palmiraichnus have an internal structure composed of an ovoid chamber and a spiral cap and a subcylindrical antechamber surrounded by a discrete wall [Celliforma lacks antechambers or constructed walls [Celliforma. This nest represents the first recording of Celliforma in Africa, discounting the nest recorded from Rusinga Island (western Kenya) [According to the classification provided by Genise 2000), only two ichnogenera have been described for solitary bee cells: lliforma . Palmiraete wall . In conted walls  and can 00, only n Kenya) , would nThe remnants of an organic substance are clear in the petrographic microscope images of the thin section. The structure –a behaviour that is seen in some species of Megachile today . This woCelliforma, containing solitary cells that are rounded at one end and truncated at the other, with smooth walls and no clear antechamber.The presence of trichomes in the lining of the cells is a feature that has not been recorded in a fossilised nest before. It must be mentioned however, that the presence of this feature is probably the result of exceptional preservation in these specimens. The majority of other nests described in the literature were not preserved to the same degree and so should trichomes originally have been present, poorer preservation would mean that they might not be visible in the fossils. These fossils may represent a new ichnospecies, but at present (until a more precise taxonomic classification is available), it seems best to classify the structures at an ichnogeneric level. The nest most closely resembles those of Celliforma nests have been recorded from Africa, and the nest cannot be placed into the criteria of any of the described ichnospecies within Celliforma from other parts of the world with any level of confidence. The ichnospecies-level classification is very complex and, at present, problematic [Celliforma but have not been named; the only named ichnospecies still believed to belong to the ichnogenus appear to be: Celliforma spirifer  [C. nuda  [C. germanica  [C. nuda and C. spirifer are both named after the distinct spiral closures of the cells–a feature that has not been identified in these cells. The type specimen for C. spirifer also averaged 270mm in length, making it considerably larger than the cells described here, whilst the cells of C. germanica are distinctly tear-shaped.No other blematic . A numbene, USA) , C. nudane, USA) , and C. Germany) . C. nudaThe structure of the cells themselves and the components of the cell lining in particular provide some features that are comparable with cells produced by extant species of bee. Ground nesting bees are prevalent throughout the world in the modern day, and different species and genera vary greatly in aspects of their nesting behaviour . A numbeMost bees line their cells with a waxy substance that is excreted from the body of the female bee. It is possible that the organic matrix that is visible in the images of the bee cell extracted from the nest being described here represents the remains of this substance. This lining is usually secreted from the Dufour’s gland with the purpose of maintaining an optimal moisture balance within the cells. However; assuming that the hair-like structures in the lining of the cell examined are indeed trichomes, it is possible to associate the trace makers with extant Megachilidae and more specifically, the sub-family Megachilinae.The Megachilinae are fairly unique amongst bees in that almost all of them utilise foreign material in the creation of the cells or partitions in their nests. There are only a few other tribes, each of which belongs to Apinae, that are known to incorporate foreign material in their nest construction . The larA large proportion of megachilids do not excavate their own burrows from scratch. Many adapt plant stems, hollow wood, or nests previously made by wasps, by creating additional burrows, making cavities smaller, and constructing a cell lining made of foreign material . Those tWith the exception of Lithurgini, the vast majority of Megachilinae use foreign materials from outside of their nests when creating cells for their larvae . The triTrachusa and Paranthidium create their own burrows using foreign material such as resin, which is occasionally mixed with soil particles, pieces of leaf, or plant hairs, amongst other materials [Bees that are best known for the inclusion of plant hairs (trichomes) in their nests belong to the tribe Anthidiini–a group more commonly known as wool carder bees that encompasses several genera and over 70 species . Althougaterials . Others aterials .Megachile have been reported to use leaf pieces, sometimes alongside chewed leaf pulp, to structure their nests, some of which are dug by the bees as burrows in the ground [Finally, bees within the tribe Megachilini are understood to incorporate leaf pieces into their nests, which are usually found in cavities, stems, manmade objects, or burrows in the ground. Most notably, members of the genus e ground .It has been shown that the use of foreign material to line larval cells creates a hydrophobic membrane and can display antimicrobial properties , 33; it Megachile) has been recorded in the form of body fossils dating back to the Paleocene [Exactly when in the evolutionary history of Hymenoptera the use of foreign materials to line nests originated is not clear, but evidence of leaf-cutting behaviour in bees  [Coprinisphaera ichnofacies have been identified at other early hominin sites in Africa. Examples include: the aforementioned site on Rusinga Island, where a fossil bees’ nest was recorded [s 2000) . This ics (2000) –all of ws (2000) ). Termit00 . Thisrecorded , and finrecorded .Coprinisphaera ichnofacies. The occurrence of the Coprinisphaera ichnofacies has a moderate to high ichnodiversity and high abundance; it is especially common in mature paleosols, and is generally indicative of savannah grasslands [Unlike at other hominin sites, excavations at the Taung Type Site have revealed all of the key ichnofamilies that signify the presence of the asslands . That duasslands  shows thasslands . This isasslands .Coprinisphaera ichnofacies are suggestive of a terrestrial herbaceous environment with arid soils, plant growth, exposure to the sun, and the presence of angiosperms [4 grasses is also a possibility [3 forbs) are however the most favoured amongst members of Anthidiini, after composites [The presence of both a bee’s nest and the iosperms . It is msibility , 44. Megsibility . Anthidisibility . Pollen mposites . A potenmposites  which inCoprinisphaera [The trace fossils of ground-nesting insects are an extremely reliable indicator of the presence of paleosols . In mostisphaera , and eggisphaera  belonginThe presence of ground-nesting bee traces provides additional evidence that the PCS deposits are of pedogenic origin. This finding has a number of wider implications for research at the Taung site. Under the cave model of the Dart and Hrdlička pinnacles, it was unlikely that many more in-situ fossils remained to be uncovered. The discovery that the hominin-bearing PCS deposits are instead of pedogenic origin has entirely different implications for future research. Hopley and colleagues (2013) found that the calcrete extends hundreds of metres beyond the pinnacles in the realms of a carbonate-rich river system . This grCelliforma. The results of the study largely reinforce the recent literature regarding the palaeoecology of the Taung Type Site, and indeed of Plio-Pleistocene southern Africa as a whole. Moreover, they strengthen the hypothesis that the Type Site deposits are pedogenic in origin, having formed on the land surface, rather than as sedimentary in-fills of caves as is the case for the majority of southern African hominin sites. Despite recent advancements in the field, the identification and implications of the presence of insect trace fossils in terrestrial deposits remains largely overlooked, especially in early hominin sites. Although common to paleosol deposits, insect traces are rarely considered in detail, yet they could offer important palaeoenvironmental insights, with the potential to reveal valuable information about hominin palaeoecology.Here the first fossil bee’s nest from South Africa has been described and the trace makers attributed to the ichnogenus


Escherichia coli, Saccharomycies cerevisiae, and Arabidopsis thaliana.The availability of high-throughput data from transcriptomics and metabolomics technologies provides the opportunity to characterize the transcriptional effects on metabolism. Here we propose and evaluate two computational approaches rooted in data reduction techniques to identify and categorize transcriptional effects on metabolism by combining data on gene expression and metabolite levels. The approaches determine the partial correlation between two metabolite data profiles upon control of given principal components extracted from transcriptomics data profiles. Therefore, they allow us to investigate both data types with all features simultaneously without doing preselection of genes. The proposed approaches allow us to categorize the relation between pairs of metabolites as being under transcriptional or post-transcriptional regulation. The resulting classification is compared to existing literature and accumulated evidence about regulatory mechanism of reactions and pathways in the cases of  Metabolism is the integrated output of transcription, post-transcriptional processes, translation and post-translational processes, and reflects the environment and changes in the availability of nutrients Stitt, . The comMicroarray and RNA-sequencing techniques can measure several thousand genes from multiple conditions and time points simultaneously  single gene and single metabolite, (ii) multiple genes and a single metabolite, (iii) single gene and multiple metabolites, and (iv) multiple genes and multiple metabolites. The first set of approaches is the simplest and aims at identifying the association for a pair of gene and a metabolite  to Y. The advantage compared to PLS is twofold: first, the orthogonal part of X can be separately investigated and secondly and more important the removal of uncorrelated variation increases the interpretation . A solution to this dimensionality problem is to focus on a subset of the data, so that the number of transcripts (metabolites) is smaller than the number of observations, which may introduce bias in the analysis  and its extensions  as well as control treatment. The metabolomic data were generated by gas chromatography-mass spectroscopy (GC-MS) and contain 192 metabolites. All measured metabolites were normalized to cell number and the chromatographic internal standard. Transcript data were measured with microarray based technique and 4,440 transcripts were detected. The normalization procedure of the transcript data is described in the Materials and Methods section of Jozefczuk et al. , nitrogen downshift (shift from glutamine two proline) and Rapamycine treatment, was included. The data set contains 256 metabolites measured with FIA-QTOF-MS and 5,716 transcripts measured with Affymetrix chips  of two variables while controlling for a third or more variables. When using a single controlling factor, one calculates the first order partial correlation. If the number of controlling factors is higher, their information is recursively removed and the second or higher order partial correlation is determined. The zero order partial correlation is the same as the Pearson correlation. The expression for recursive calculation of partial correlation between the variables Z ∈ V.where The combination of partial correlation and PCA allows the calculation of the two approaches Transcriptional dependent Partial Correlation (TPC) and Post-transcriptional dependent Partial Correlation (PPC).p PCs of the transcript data and use them as controlling variables for the partial correlation for each combination of metabolites. The number of PCs to choose was investigated based on the Broken-Stick model  of the trancriptomic data, (2) we determine all metabolite pairs which have a significant Pearson correlation coefficient, (3) we determine all metabolite pairs which have non-significant partial correlation upon removal of the controlling variables, i.e., the p PCs from step (1), above, and (4) for the pairs of metabolites in the sets obtained from (2) and (3), we select those that show a significant difference between their Pearson correlation and partial correlation values. The reason for such construction of the approach is the following: if the removal of the significant PCs leads to a non-significant partial correlation between two metabolites, their association was due to transcriptional regulation. As the significant PCs capture most of the transcriptional effects, by finding the partial correlations we remove most of the transcriptional influence on the association between the two metabolites. The statistical tests ensure that the consideration of the significant PCs indeed break the significant association between metabolites and that the difference between the values is significant. To determine statistical significance we rely on permutation tests .Our first approach, termed TPC, aims at identifying pairs of metabolites whose association is related to transcriptional regulation. The approach is composed of four steps: (1) We calculate the first The second approach, termed PPC, follows a similar methodology. Again, the significant PCs from the transcriptomics data set are used as control variables in the partial correlation analysis for pairs of metabolites. In contrast to the TPC approach, we select those pairs of metabolites that are significantly associated upon removal of the significant PCs. Similar to TPC, we select those pairs of metabolites whose partial correlations show significant difference from the values of the respective Pearson correlation coefficient. The approach is based on the premise that if correlation remains upon removal of the transcriptional effect, the observed association is due to post-transcriptional regulation of the two metabolites. The significant difference to the observed Pearson correlation is employed to ensure that the observed partial correlation is due to post-transcriptional effects.E. coli, S. cerevisiae and A. thaliana . To this end, we investigated the number of significant PCs for each of the available data sets. The Kaiser-Guttman approach suggested the use of three PCs in each of the three data sets, whereas the Broken-Stick model suggest the usage of only one PC for E. coli and A. thaliana and two for S. cerevisiae. The PA approach confirms the one PC for E. coli and two for S. cerevisiae. However, the approach estimates two significant PCs for the A. thaliana data set. Overall, we found between one and three significant PCs, depending on the approach and data set , containing the levels of 192 metabolites and 4,400 genes over five conditions. Employing the TPC resulted in 87 metabolite pairs under transcriptional control , “downstream” , “upstream”  and “parallel” . The assignment of metabolites into on of the four categories is done by evaluating the dynamic dependence of metabolite and transcript pairs over time and the association of each metabolite with a specific set of genes regulated by TORC1, called “representation of TOR genes.” Both features are combined in a Bayesian inference framework approach to calculate the probability for each metabolite to belong to one of the four motifs. If the probability is above 50%, the metabolite is assigned to that particular motif. Eight metabolites were assigned to the downstream motif, eight metabolites to the upstream motif and eight metabolites to the parallel motif.So far we were able to identify the prevailing regulatory mechanism between identified pairs of metabolites. However, our comparison focused on a broad literature comparison, but did not compared our approach directly with a comparable method capable of integrating transcriptomic and metabolomic data into a combined analysis. Therefore, we chose to complement our study with a comparison with the results obtained in Oliveira et al. . The stuE. coli, we observed no change in the sign of correlation between Pearson and partial correlation . Within this data set, we found 295 transcriptional associated metabolite pairs with the TPC approach  and tyrosine . Tryptophan was not reported within the uniquely identified metabolite pairs. At the same time shikimate is positively correlated to pyruvic acid , fructose , glucose  and sucrose . We therefore observed that metabolites upstream of shikimate (sugars) were positiveley correlated, while downstream metabolites were negatively correlated. The pathway is partly feedback regulated meaning that the end products (amino acids) inhibit their production which explains the negative correlation. The sugars were positively correlated to shikimate as they are potential precursors (Tzin and Galili, E. coli we found more pairs with both approaches. The correlations of the TPC approach was higher than in E. coli. A similar situation was observed for the PPC approach. We observed more sugars and sugar derivatives in E. coli, whereas amino acids were mostly found with high positive correlation.In comparison In this study we proposed two approaches for a combined investigation of metabolic and transcriptomic data. The two proposed approaches are based on the concept of removing transcriptional information from metabolomic data, allowing us to categorize pairs of metabolites into transcriptional or post-transcriptionally regulated. The developed approach TPC allows the identification of transcriptionally regulated metabolites through a modified partial correlation approach using PCs of the transcriptomic data as controlling variables. The second approach, PPC, is based on a similar concept and it allows the identification of post-transcriptional regulation between pairs of metabolites.The commonality of the investigate data sets is their focus on the change of central metabolites after perturbation or changing environmental conditions. It has been shown that in microorganisms the majority of primary metabolites are mainly regulated on the enzymatic level through feedback inhibition (Sanchez and Demain, While we observed a differentiation into pairs found by TPC and PPC, the detected partial correlation in each approach did not differ strongly from the found Pearson correlation (see Figure In the recent work of Bradley et al. , they reA potential application for our proposed approaches is metabolic engineering. Metabolic engineering aims at enhancing certain important pathways which leads to an overproduction of a metabolite of interest Bailey, . A frequOverall, we present here two approaches named TPC and PPC for investigating the prevalent regulatory mechanism of metabolite pairs. To our knowledge it is the first time that partial correlation is used to remove all transcriptional information from a metabolomic data set, removing not just the effect of a set of genes, but the majority of transcriptional regulation. This novel investigation methods will help to elucidate the complex regulatory mechanism of metabolites while employing well known and established statistical methods.ZN conceived the project and wrote the article with contribution of KS. KS performed the analysis and analyzed the data.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


R2 via cross-validation to compare fitting performance between area-separated and area-combined standardizations, and (3) system-based testing to explore the consistency of the standardized CPUEs with auxiliary data in the PBF stock assessment model. The last stage of evaluation revealed high consistency among the data, thus demonstrating improvements in data reconstruction for estimating the abundance index, and consequently the stock assessment.Catch-per-unit-effort (CPUE) is often the main piece of information used in fisheries stock assessment; however, the catch and effort data that are traditionally compiled from commercial logbooks can be incomplete or unreliable due to many reasons. Pacific bluefin tuna (PBF) is a seasonal target species in the Taiwanese longline fishery. Since 2010, detailed catch information for each PBF has been made available through a catch documentation scheme. However, previously, only market landing data with a low coverage of logbooks were available. Therefore, several nontraditional procedures were performed to reconstruct catch and effort data from many alternative data sources not directly obtained from fishers for 2001–2015: (1) Estimating the catch number from the landing weight for 2001–2003, for which the catch number information was incomplete, based on Monte Carlo simulation; (2) deriving fishing days for 2007–2009 from voyage data recorder data, based on a newly developed algorithm; and (3) deriving fishing days for 2001–2006 from vessel trip information, based on linear relationships between fishing and at-sea days. Subsequently, generalized linear mixed models were developed with the delta-lognormal assumption for standardizing the CPUE calculated from the reconstructed data, and three-stage model evaluation was performed using (1) Akaike and Bayesian information criteria to determine the most favorable variable composition of standardization models, (2) overall  The catch rate or catch per unit effort (CPUE) expresses the number of fish caught with a certain amount of fishing effort; it is frequently used as an index of relative fish abundance . Catch aThunnus orientalis) are highly migratory in the temperate areas of the North Pacific Ocean (NPO), with occasional appearances in the Southern Pacific Ocean near Australia and New Zealand . Ex. Ex28]. Before the analyses were conducted, data were aggregated in advance on a trip basis, as followed for the Japanese PBF CPUE standardization , becausePBF are infrequent in the catch; therefore, the CPUE data was composed of a high proportion of zero observations, which may violate the model assumptions and jeopardize the integrity of the inferences if not properly modeled . This stA small number of explanatory variables were considered in the study because of the limited information available. Therefore, a simple forward (increasing variables) then backward (decreasing variables) method was used for determining the variables to be included in the model. All three explanatory variables  were initially included in both models and were retained in the final models by using the backward method. Furthermore, first-order interactions of the explanatory variables were considered for both models and were determined using the forward method. The interactions of year and other categorical variables (month and vessel size) were considered random variables ,44. The R2 value estimated through cross-validation and bootstrap procedures. The last stage involved investigating which of the three standardized CPUE series had the most favorable consistency with other data associated with PBF resources , and thus could be considered as the abundance index of Taiwanese fishery, by applying the CPUE data to the PBF stock assessment model.Three methods corresponding to the three categories methods reported by Hinton and Maunder  were useFirst, the AIC and BIC  obtainedR2 value was estimated through cross-validation and bootstrap procedures  with the predicted probability of zero from the ZPM obtained in step (ii). The value was then multiplied with the predicted CPUE obtained from the second component of the delta approach (PCM) obtained in step (ii). Thus, pairs of observed and predicted CPUE for all data in the validation set could be obtained and separated for each of the three sets of CPUE standardization models. The R2 value was then calculated from the pairs of observed and predicted CPUE for the area-separated model (combining pairs of data from the CPUE models for the southern and northern fishing grounds) and the area-combined model.The second subset was considered the validation set and was used to calculate ivR2 for the area-separated and area-combined models.Steps i–iii were repeated 200 times to obtain the mean and 95% confidence intervals of http://nft.nefsc.noaa.gov/Stock_Synthesis_3.htm. The third stage involved system-based testing by applying the final standardized CPUE series from the first stage to the latest PBF stock assessment model. The model was established by the ISC PBF Working Group in 2016 , which wTaiwanese longline southern CPUE and Japanese longline CPUE were fitted in the model.Taiwanese longline area-combined CPUE and Japanese longline CPUE were fitted.Taiwanese longline southern and northern CPUE and Japanese longline CPUE were fitted.Only the Japanese longline CPUE was fitted.Only the Taiwanese longline southern CPUE was fitted.Monte Carlo simulation has numerous applications in fisheries . The appAfter the distribution assumption was determined, the simulation was tested on the 2004–2006 data to compare the actual fish number distribution with the simulated one, and the KS test revealed no statistical differences between them . The Monte Carlo simulation was subsequently applied to estimate the catch number for each year during 2001–2003.This task essentially defined the suitable fishing day classification criteria from the VDR data of the fishery. The application of the three fishing day classification approaches to the 2010–2015 VDR data revealed the performance statistics of the criteria (SSS), indicating the following maximum values: a speed of 3 knots for the within-day vessel speed approach , a dailyAn example plot of the tracks of nonfishing and fishing definitions based on the optimal criterion  revealedStudies exploiting existing independent geolocation data to develop fishing day classification criteria and then deriving fishing effort data from the criteria have become popular since the mid-2000s ,53,54. TIn addition to the VMS, Chang  used anoThis study applied the approaches for developing fishing day criteria that were recommended by Chang and Yuan  and usedThe direction change approach showed the optimal performance among the three approaches, with much higher SSS and lower DSS. A longline operation occurs in two parts: hook deployment and, after a short break for bait soaking, hook retrieval. After hook deployment, the vessel may start the hook retrieval operation from the position of the end of the deployment  or return to the beginning position of the deployment (starting from the earliest deployed hooks). Irrespective of the type, the vessel must change the head direction substantially; this can be used for developing fishing day classification criteria. This approach was not previously tested on a distant-water longline fishery operating in a large area . The dirFishing days are typically calculated from logbooks. In many cases, at-sea days are used or proposed to represent fishing days when logbooks are insufficient ,58. HoweR2 > 90%. The coefficients were subsequently applied to the vessel trip data for 2001–2006 to estimate the fishing days from at-sea days by the vessel size and port.An analysis of variance (ANOVA) was performed to investigate the relationship between the response variable of the fishing days estimated from the VDR data and the explanatory variables of the at-sea days calculated from the vessel trip data, by the vessel size and port, for 2007–2015 , S5 FileVessels leaving and entering Taiwanese ports must register for inspection by the CGA for security purposes because of the special antagonistic relationship between Taiwan and China . The datThe aforementioned procedures addressed the incomplete data situation and made the CPUE of the whole series available for deriving a standardized index. By using the backward method for selecting primary explanatory variables and the forward method for selecting first-order interaction variables, the optimal variable combinations for the area-separated and area-combined standardizations were decided by the smallest AIC and BIC values , the firThe GLM-based method with lognormal assumption for CPUE standardization was used because this method is commonly used for highly migratory species  ,40,46 anRandom effects are typically introduced into CPUE standardization models to address interactions between the year and other categorical variables that result in random changes in the population . In thisTwo important factors should be considered in the model variables but cannot be included in the standardization model because of insufficient information are the target and habitat effects. PBF is a seasonal target species for the Taiwanese longline fishery. However, in seasons when the PBF catch is low, registered PBF vessels might shift to fish for other tuna species to increase profits. Therefore, target information is useful in the standardization process. However, because no logbook data were available for the period before 2010, a complete series of target information could not be obtained. The fish availability was low in all PBF vessels, and 99% of the trips caught 0–2 fish per day; therefore, we expect the effect to be insubstantial. Lynch et al.  concludeWhen statistical models are fitted, adding parameters may increase the likelihood of overfitting; both AIC and BIC aim to address this problem by introducing a penalty term for the number of parameters to deal with the trade-off between the goodness-of-fit and complexity of the model . HoweverR2 value directly from the observed and model-predicted CPUE. In this study, considering the difficulty in using the AIC and BIC for optimal model selection, the overall R2 value could be a more straightforward approach for model performance comparison. To avoid the overfitting effect, the second model evaluation method used cross-validation and bootstrap procedures to determine the overall R2 value. The results revealed that area-separated standardization yielded higher overall R2  than did the area-combined one , suggesting that area-separated standardization has a more favorable fitting performance than the area-combined one.An alternative approach for comparing the model fit is to calculate the Furthermore, the CPUE series of the northern fishing ground revealed a different relative abundance trend than did that of the southern fishing ground, although the combined CPUE series seems to be intermediate to that of both series, particularly in the most recent 3 years . Most hiThe third model evaluation method is system-based testing through application of the three standardized CPUE series  to the PThis system-based testing method examines the diverse available information beyond catch and effort data and is based on the consistency of standardized CPUE estimates with auxiliary information. The results clearly support the CPUE series of the southern fishing ground according to the fit of the assessment model to that data as well as its consistency with the other data in the assessment model, such as the Japanese longline other fisheries’ CPUEs and size composition data. In particular, the comparison of scenarios 4 and 5 indicated that the size composition data support the CPUE series of the Taiwanese southern fishing ground over the Japanese longline CPUE. Contrastingly, the assessment model does not support the information derived by the CPUE series of the northern fishing ground. This fishing ground has been developed in recent years. Therefore, some factors might still be related to PBF catchability and should be explored and considered in the standardization procedures, such as the finer spatial and temporal effect on PBF availability from the interaction of PBF migration and fishing gear position.The RMSE for the fit to the Taiwanese CPUE estimated in this study was 0.27, whereas a previous assessment conducted in 2014 revealed a higher RMSE (0.41), which was estimated and standardized using different data and methods  . These rBased on these results and qualitative consistency in the fishing activity and the fish size caught in each fishing ground within the PBF distribution range, the system-based testing demonstrated that data reconstruction has improved the abundance index estimation and, consequently, has addressed the concerns of data inconsistency indicated in the previous PBF stock assessment. The PBF stock assessment model is an integrated stock assessment model which has become the dominant method for assessing tuna stocks. The model represents scientific understanding of the dynamics of the population in equations that define how the population and its structure changes over time and uses all available information about the population . TherefoS1 FileData for 2015 are preliminary.(XLSX)Click here for additional data file.S2 File(XLS)Click here for additional data file.S3 File(XLSX)Click here for additional data file.S4 File(XLSX)Click here for additional data file.S5 File(XLSX)Click here for additional data file.S6 File(XLSX)Click here for additional data file.


Speech production impairment is a frequent deficit observed in aphasic patients and rehabilitation programs have been extensively developed. Nevertheless, there is still no agreement on the type of rehabilitation that yields the most successful outcomes. Here, we ran a detailed meta-analysis of 39 studies of word production rehabilitation involving 124 patients. We used a model-driven approach for analyzing each rehabilitation task by identifying which levels of our model each task tapped into. We found that (1) all rehabilitation tasks are not equally efficient and the most efficient ones involved the activation of the two levels of the word production system: the phonological output lexicon and the phonological output, and (2) the activation of the speech perception system as it occurs in many tasks used in rehabilitation is not successful in rehabilitating word production. In this meta-analysis, the effect of the activation of the phonological output lexicon and the phonological output cannot be assessed separately. We further conducted a rehabilitation study with DPI, a patient who suffers from a damage of the phonological output lexicon. Our results confirm that rehabilitation is more efficient, in terms of time and performance, when specifically addressing the impaired level of word production.


In recent years, the use of aluminum alloy has tended to increase for building lightweight automobiles to reduce their automotive weight, which is helpful to save energy and protect the environment. In order to join aluminum alloy, a flat-clinching process combined with material forming technology was investigated to join aluminum alloy sheets using an experimental and a numerical method. Al1060 was chosen as the material of the sheet, and DEFORM-2D software was used to build the numerical model. After the numerical model was validated by the experimental results, the influences of punch diameter and holder force on the materials deforming behavior of the clinched joint were analyzed using the numerical model. Then, the material flow, joining ability, and joining quality were investigated to assess the clinched joint. The results showed that an increase in punch diameter could give rise to an increase in neck thickness and interlocking length, while an increase in blank holder force induced a decrease in interlocking length and an increase in neck thickness. The joining quality could be increased by increasing the forming force. It can be concluded that a clinched joint has better joining quality for joining light-weight sheets onto automotive structures. Events such as the increase in greenhouse gases and the oil crisis in the 1970s have become catalytic factors to a requirement for reducing energy consumption ,3, whichTo build lightweight structures of automobiles, the use of aluminum alloy has tended to increase with its high specific properties ,7. The jPrevious studies have investigated the process parameters of the conventional clinching process and the geometrical parameters of conventional clinching tools. Lambiase  investigThe conventional clinching process is applied to join many materials, such as copper alloy , titaniuMany experimental studies have been conducted to investigate the conventional clinching process. These experiments usually consume too much time and too many resources, which make the experiments expensive and complicated. In order to solve these problems, finite element simulation is frequently used to investigate the conventional clinching process using different simulation software . Lambiase and Di Ilio  developeIn the conventional clinching process, a punch is applied to compress two or more sheets, which may form an exterior protrusion on one side of the sheet. With the redundant protrusion, it is not suitable for a clinched joint to be applied in visible places. This has become one of the main problems affecting the wide use of a clinched joint. Chen et al.  proposedIn the present work, a flat-clinching process combined with material forming technology was investigated using an experimental and a numerical method. The numerical model was built using DEFORM-2D software, and Al1060 was chosen as the material of the sheet. The numerical model was validated by the experimental result, which proved that it was feasible to use the numerical model for predicting the outcome of a flat-clinching process with an accurate result. The influence of punch diameter and holder force on the materials deforming behavior of the clinched joint was analyzed using this numerical model. Then, the material flow, joining ability, and joining quality were investigated to assess the clinched joint. It is proved that a clinched joint has better joining quality for joining Al1060 sheets.A two-dimensional axisymmetric numerical model was built in the finite element simulation software DEFORM-2D. Compared with a three-dimensional numerical model, the two-dimensional axisymmetric numerical model can save a lot of time because it uses a small number of elements.In recent years, aluminum alloy has been widely applied to build mechanical structures because of its good forming abilities. Al1060 was taken as the sheet material in this finite element simulation. The thickness of the sheet was set to 2.0 mm. The Poisson’s ratio of the Al1060 is 0.33; its Young’s modulus is 54.5 GPa; and its tensile strength is 117.9 MPa. The sheet material was modeled as a plastic body, while the flat-clinching tools were modeled as rigid bodies. Quadrilateral elements were used to mesh the sheets, involving 3938 elements with 4140 nodes in the numerical model. The flat-clinching tools consist of an upper punch, a fixed flat anvil, and a blank holder. The main geometrical parameters of the numerical clinching tool model are shown in The Coulomb friction law is suitable for a numerical simulation of the clinching process. Wen et al. , He et aThe upper punch diameter, clinching force, and blank holder force have an important influence on the material flow, joining quality, and deforming characteristics of a clinched joint. The variation of punch diameters needs many different upper punches, which may increase the cost of the experimental study. So, the influence of upper punch diameters on the materials deforming behavior was investigated by finite element simulation. The different punch diameters of 4.3, 4.7, 5.1, 5.5, and 5.9 mm were adopted in the numerical simulation. The clinching force can be varied by controlling the clinching machine, which is easy to operate by an experimental method. The holder force applied to the blank holder was generated by various combinations of disc springs as shown in As shown in X), interlocking length (st), and neck thickness (nt), are shown in A mechanical structure is produced to clinch the sheets. Thus, the geometrical shape determines the joining ability and the quality of the flat-clinched joint. The shape’s main parameters, including the bottom thickness . Joining dies made by molded steel are shown in To assess the joining reliability of the joint, a tensile test (testing tensile strength) and a shearing test (testing shearing strength) are always carried out. Coppieters et al.  pointed Forming force is one of the key parameters affecting the joining reliability of a clinched joint ,38,39. TBefore using the numerical model to carry out the finite element simulation, a numerical model validation is reported to discuss the accuracy of the numerical model. The main geometrical parameters, such as neck thickness and interlocking length, affect the joining quality of a clinched joint . Thus, aThe experimental results of the neck thickness and interlocking length were measured by sectioning the clinched joints along the axial plane. Five repetitions for each joining condition were conducted to obtain the mean value.The comparison of the numerical results and experimental data is shown in Punch diameter is one of the key parameters in the flat-clinching process. The influence of punch diameter on the materials deforming behavior is investigated in this section. The shapes of the deformed zones resulting from the numerical simulations with different punch diameters are shown in The neck thicknesses and interlocking lengths of joints with different punch diameters are shown in As shown in Different clinched joints were formed using different blank holder forces. Other geometrical parameters of the clinching tools and the process parameters of the flat-clinching process were kept consistent. Different combinations of disc springs were used to generate different blank holder forces. The neck thicknesses and interlocking lengths of the clinched joints with different combinations of disc springs are shown in A die holder was applied to hold the Al1060 sheets on a flat anvil. As shown in The velocities of point ‘A’ and point ‘B’ during each performance of the flat-clinching process with different combinations of disc springs are shown in In addition, as shown in The joining process consists of four stages as shown in Firstly, the blank holder fixed two Al1060 sheets on the flat anvil. Then, a forming force was applied to the punch to compress the Al1060 sheets. The sheet materials would flow downward under the punch. With the limitation of the flat anvil, the material flowing downward would turn to flow radially. The die holder gave a limit to the radial material flow, which resulted in the upward flow of the sheet material. The upward material would be collected in the gap between the die holder and punch. During the third stage, the lower sheet material was also pushed to flow opposite to the punch direction. During the last stage, a mechanical joining interlock was formed, which clinched the two Al1060 sheets together. The flat anvil restricted the material flow downward, which resulted in a flat sheet surface with no exterior protrusion.As shown in Tensile strength is always used to assess the joining reliability of the joint . The staEnergy absorption in the process of failure is another important evaluation criterion for clinched joints . A higheThe energy absorption of a joint was increased by increasing the joining force. The joint formed by a joining force of 90 kN had the highest energy absorption in the test, while the clinched joint formed by a joining force of 60 kN had the lowest energy absorption. The joint with a joining force of 90 kN had the longest displacement and highest strength in the tensile test, which gave rise to the highest energy absorption.(1)A punch with a larger diameter could push more materials of the bottom parts  to flow to the neck and interlock areas, which can give rise to an increase in the neck thickness and interlocking length. With a larger neck thickness and interlocking length, the joining quality of the clinched joint could be improved.(2)An increase in blank holder force induces a decrease in interlocking length and an increase in neck thickness. With a lower holder force, more material would flow in the radial direction without the restriction of the blank holder, which could generate a smaller interlocking length. In the flat-clinching process, a higher holder force is required to restrict the material flow.(3)A mechanical joining interlock is formed during the joining process. The material flow in the joining process consists of four stages: (a) local deforming; (b) drawing; (c) backward extruding; and (d) interlock forming.(4)For the joints formed by a joining force of 50 kN, there is no mechanical interlock generated, while for the joints formed with the joining forces of 60, 70, 80, and 90 kN, a mechanical interlock was formed because of the plastic deformation. Joining reliability could be improved by increasing the forming force.In the present work, a flat-clinching process combined with material forming technology was investigated using experimental and numerical methods. After the numerical model was validated with the experimental results, the influence of punch diameter and holder force on the materials deforming behavior of the clinched joint was investigated. Then, the material flow, joining ability, and joining quality were investigated to assess the clinched joint. The main conclusions of the present work can be drawn as follows: