Multibody models built in commercial software packages, e.g., ADAMS, can be used for accurate vehicle dynamics, but computational efficiency and numerical stability are very challenging in complex driving environments. These issues can be addressed by using data-driven models, owing to their robust generalization and computational speed. In this study, we develop a deep neural network (DNN) based model to predict longitudinal-lateral dynamics of an autonomous vehicle. Dynamic simulations of the autonomous vehicle are performed based on a semirecursive multibody method for data acquisition. The data are used to train and test the DNN model. The DNN inputs include the torque applied on wheels and the vehicle’s initial speed that imitates a double lane change maneuver. The DNN outputs include the longitudinal driving distance, the lateral driving distance, the final longitudinal velocities, the final lateral velocities, and the yaw angle. The predicted vehicle states based on the DNN model are compared with the multibody model results. The accuracy of the DNN model is investigated in detail in terms of error functions. The DNN model is verified within the framework of a commercial software package CarSim. The results demonstrate that the DNN model predicts accurate vehicle states in real time. It can be used for real-time simulation and preview control in autonomous vehicles for enhanced transportation safety. Recent advances on neural networks have led to dramatic progress in solving many complex engineering problems, and demand for such solutions is increasing with time. A wide range of topics starting from dynamics modelling, intelligent control to transportation research is being handled by neural networks ,2,3. ThiIn recent years, a number of contributions have been reported in the field of vehicle coupling dynamics modelling and control by using neural networks. Kumarawadu et al. proposed a neural network adaptive control approach for longitudinal-lateral dynamics of vehicles for highway applications. The results in the face of parameter uncertainties indicated the stability and robustness of the control algorithm . Melzi eThe literature survey indicates that DNNs can be efficiently applied in vehicle dynamics modelling and control . In mostTo address these issues, a data-driven modelling method based on real-time data acquisition and DNNs provides an efficient solution. In this study, a DNN model for predicting longitudinal-lateral dynamics of a vehicle is presented. An efficient semirecursive multibody method that performs real-time simulations is used to capture vehicle key characteristics. The highlights of this study lies in two aspects. The first aspect consists of the use of an efficient semirecursive multibody formulation to acquire training and testing data. The multibody model can deal with the nonlinearities of vehicle systems more accurately than the simplified or decoupled models. The second aspect consists of a DNN modelling approach to predict longitudinal-lateral dynamics of a vehicle. Different applied torques and initial speeds over a wide range are used to imitate various kinds of on-road driving situations. The DNN model is verified by using the results obtained from a commercial software package CarSim. Further, the DNN models with different hidden layers and sample size are investigated in terms of accuracy and efficiency. The widely used backpropagation neural network and radical basis function neural network are employed as references to tell the advantages of the DNN model used to predict the vehicle states.The rest of the study is organized as follows. In In this section, we used an efficient semirecursive multibody method to develop a vehicle dynamics model. The equations of motion of the closed-loop vehicle systems can be formulated in terms of independent relative (joint) coordinates. They can be expressed as ,32: (1)RThis formulation is called the double-step semirecursive multibody formulation, which was proposed by Javier García de Jalón and his coworkers ,34. Althegrators ,36. Low-The training and testing sets of vehicle dynamics data can be obtained in real time based on the semirecursive multibody formulation. Thus, we aim to collect historical data that include the torque applied on wheels, the vehicle’s initial speed, the longitudinal driving distance, the lateral driving distance, the final longitudinal velocities, the final lateral velocities, and the yaw angle to build a lateral-longitudinal dynamics model by using a DNN approach. The capacity of the DNN modelling approach to capture the key vehicle states is explored, and verified based on the results obtained from a commercial software package CarSim. The data-driven vehicle model is developed offline via DNN approach and used online for preview control in autonomous vehicles. The computational burden of the data-driven model is much smaller than that of the vehicle multibody models, e.g., ADAMS model, CarSim model, and semirecursive multibody model. Furthermore, the data-driven model avoids the numerical integration process of vehicle multibody models, and it leads to robust generation of the vehicle states.DNNs can learn and store relationships between the input and output neurons. Mathematical theory has proved that DNNs can approach any nonlinear continuous functions with high precision. A general DNN framework is described in n-th layer, respectively. m denotes the sample size in the training set, and n denotes the number of hidden layers and input layer,The DNN training process is composed of forward propagation of inputs and backpropagation of errors. The input matrix, the weight matrices, and the bias matrices in forward propagation are described as:i-th layer, respectively, and i-th activation function. Note that The forward propagation steps are expressed as:e denotes the network’s cost, The mean square error (MSE) was selected as the loss function to measure the differences between the DNN results and sample results before backpropagation. By implementing backpropagation, the weight matrices and bias matrices were updated to minimize the loss function. In this study, L2 regularization was used to calculate the loss function to avoid overfitting. The MSE, the L2 regularization, and the loss function are expressed as :(8)MSE=YThe adaptive moment estimation optimization (Adam) algorithm was used in the backpropagation process. It can be regarded as a combination of the root mean square prop (RMSProp) algorithm and adaptive gradient (AdaGrad) algorithm . The maiBecause the moving exponential weighted average causes relatively large errors to the initial value at the beginning of the iteration, the AdaGrad and RMSProp parameters were updated by using the following expressions:These parameters were corrected in each training iteration. The weight matrices and bias matrices, in turn, were updated based on the Adam algorithm:In this section, we used the semirecursive multibody formulation described by Equation  to obtaiWe obtained critical vehicle states by introducing 500 different initial speeds and driving torques at defined steering angles. Those vehicle data, including the longitudinal distance and speed, the lateral distance and speed, and the yaw angle, denote the longitudinal-lateral dynamics of the autonomous vehicle. The range of the initial speed is 15 m/s to 45 m/s. The range of applied torque is −500 Nm to 500 Nm.x represents the value of the samples, We randomly selected 500 samples and divided them into a training set with 450 samples and a testing set with 50 samples. The training and testing sets were used to train and test the DNN model. Furthermore, the training data were processed to eliminate the inconvenience caused by the magnitude of values. It can avoid small weight matrices, which may cause numerical instability during the integration. We used Z–score standardization to process the data, and the equation is described as:There are two widely used methods to update the optimization parameters in DNNs. The first is batch gradient descent method that calculates all samples . Its comBy training the datasets, we built a DNN model to predict longitudinal-lateral dynamics of the autonomous vehicle. The DNN model for vehicle longitudinal-lateral dynamics is described in v and M in z in The inputs of the DNN model include the initial speed and applied torque of the vehicle, as described by When the DNN training is completed, the testing set is used for model test and verification. The MSE (loss function) of the dataset needs to be calculated. The DNN model can be used for the prediction and preview control if the MSE is very small. The procedure of DNN modelling for vehicle’s lateral-longitudinal dynamics is described in In this section, the DNN model of the vehicle was developed. The DNN results were described and compared with the results of the multibody model. We used box plots, described in Two conclusions can be drawn based on the absolute percentage error analysis and results comparison.-The DNN results of the longitudinal distance and speed fit the multibody model results well. The median, mean, and maximum absolute percentage errors are less than 1%.-The lateral distance and speed, and yaw angle predicted by the DNN model fit the reference results (multibody model results) well. The median, mean, and maximum absolute percentage errors are less than 3%.i-th value of the reference results, i-th value of the DNN results, i-th mean value of the DNN results, and n denotes the number of groups, which equals 7701 in this study. The five evaluation metrics were used to quantify model accuracy. The smaller the first four metrics are, the more accurate the DNN model is. However, for In this section, the accuracy of the DNN model was evaluated quantifiably by error functions. We also investigated its computational efficiency. The accuracy evaluation metrics include the mean absolute error (MAE), the mean absolute percentage error (MAPE), the maximum absolute error (ME), the root mean square error (RMSE), and The results of accuracy analysis are described in To better visualize the high accuracy of the proposed DNN modelling method, the widely used backpropagation (BP) neural network and radical basis function (RBF) neural network were used as references for comparison. The solution accuracy was investigated in term of evaluation metrics, e.g., ME, MAE, MAPE, RSME, and To investigate the computational efficiency of the DNN model, the CPU time of DNNs with different numbers of hidden layers were compared. The simulation times of the multibody model with different time-steps were measured for references. The results are shown in detail in It is easy to observe in Furthermore, to investigate the effects of dataset size on the accuracy of DNN model, different numbers of samples were used to train the DNN model. The MAPE was calculated to investigate how large the data size can predict the longitudinal-lateral dynamics effectively. It can be seen from In this section, the results obtained from the commercial software package CarSim were used to verify the effectiveness of the DNN model. It is noteworthy that vehicle simulations with the same parameters were performed in CarSim. As presented in above figures, the predicted results via DNNs coincide with the numerical results of the software package CarSim. The DNN results also are consistent with the multibody model results. The driving situations imitate the vehicle’s decelerations and accelerations during a double-lane change maneuver with a wide range of initial speeds. As a result, the effectiveness of the DNN model to predict the vehicle’s longitudinal-lateral dynamics was verified. The DNN model can be trained offline according to a large quantity of vehicle data and used for the preview control in vehicles to improve the handling, performance, and road safety. This DNN model is particularly very useful for the preview control in autonomous vehicles for enhanced transportation safety. Note that the DNN model was developed for a specific vehicle system. Useful DNN models of different vehicle systems can be developed based on a similar modelling procedure.In this study, a DNN model to predict the longitudinal-lateral dynamics of an autonomous vehicle was developed. A semirecursive multibody formulation was used to capture the vehicle characteristics. An algorithm to obtain the data of longitudinal-lateral dynamics and a DNN modelling approach were presented. It is significant that massive parallel computations can be performed by using GPUs that are especially suitable for the large volumes of data in autonomous vehicles. To verify the DNN model, the testing data were randomly selected for accuracy evaluation. Error functions also were used to quantifiably analyze the model accuracy. The computational efficiency and accuracy were investigated for neural networks with different structures. Finally, the results from a commercial software package CarSim were used to validate the proposed DNN model. The results indicated that the DNN model can predict the vehicle’s lateral-longitudinal dynamics accurately and efficiently. The DNN model used for modelling lateral, longitudinal, and vertical coupling dynamics will be investigated in the future. Overall, a DNN-based methodology was presented to predict the vehicle’s lateral-longitudinal dynamics in real time. It is suitable for the preview control of autonomous vehicles in complex environment to enhance the transportation safety.


In recent years, cinematography and other digital content creators have been eagerly turning to Three-Dimensional (3D) imaging technology. The creators of movies, games, and augmented reality applications are aware of this technology’s advantages, possibilities, and new means of expression. The development of electronic and IT technologies enables the achievement of a better and better quality of the recorded 3D image and many possibilities for its correction and modification in post-production. However, preparing a correct 3D image that does not cause perception problems for the viewer is still a complex and demanding task. Therefore, planning and then ensuring the correct parameters and quality of the recorded 3D video is essential. Despite better post-production techniques, fixing errors in a captured image can be difficult, time consuming, and sometimes impossible. The detection of errors typical for stereo vision related to the depth of the image  during the recording allows for their correction already on the film set, e.g., by different scene layouts and/or different camera configurations. The paper presents a prototype of an independent, non-invasive diagnostic system that supports the film crew in the process of calibrating stereoscopic cameras, as well as analysing the 3D depth while working on a film set. The system acquires full HD video streams from professional cameras using Serial Digital Interface (SDI), synchronises them, and estimates and analyses the disparity map. Objective depth analysis using computer tools while recording scenes allows stereographers to immediately spot errors in the 3D image, primarily related to the violation of the viewing comfort zone. The paper also describes an efficient method of analysing a 3D video using Graphics Processing Unit (GPU). The main steps of the proposed solution are uncalibrated rectification and disparity map estimation. The algorithms selected and implemented for the needs of this system do not require knowledge of intrinsic and extrinsic camera parameters. Thus, they can be used in non-cooperative environments, such as a film set, where the camera configuration often changes. Both of them are implemented with the use of a GPU to improve the data processing efficiency. The paper presents the evaluation results of the algorithms’ accuracy, as well as the comparison of the performance of two implementations—with and without the GPU acceleration. The application of the described GPU-based method makes the system efficient and easy to use. The system can process a video stream with full HD resolution at a speed of several frames per second. Stereopsis—the natural ability of humans and animals to see the 3D world ,2—still The application of current technologies such as high-resolution cameras and high-performance software tools for video analysis and post-processing causes stereoscopic films to be more realistic and impressive. The 3D displaying techniques previously available only in cinemas  are now implemented in standard TV sets, so 3D content is readily and commonly available. However, the production of 3D video is a complex process, more challenging than in the case of 2D. People responsible for creating and supervising the 3D video recording and post-production need to know the new issues and restrictions coming together with the transition from 2D to 3D content.The most important new aspect related to 3D video quality assessment is visual comfort. An incorrect 3D image may not only be aesthetically poor, but may also harm the viewer’s well-being. The key sources of visual fatigue are vertical disparity, vergence-accommodation conflicts, stereoscopic window violation, or exceeding the horizontal disparity limits . Many ofThe image registration process is the crucial stage in creating 3D content that affects the quality of the final material. The 3D quality assurance and control should start from the very beginning of the movie production—from the raw image recording. Even though the raw data go through post-production, where some errors can be fixed or minimised, it is essential to ensure that it is high quality and error free. Its further processing and improvement can be costly and time consuming, and sometimes, it can be even impossible to fix some errors in the raw input image.Electronic and IT tools for monitoring the quality, especially the disparity limits, can significantly facilitate the work of a film crew and enable quick and reliable detection of errors during 3D video recording. Exemplary systems supporting the recording crew were proposed in ,8,9. TheThe system presented in this paper is dedicated to supporting the film crew, especially stereographers, in 3D video quality control and assessment see . It can Support during calibration of the stereoscopic rig to precisely adjust the optical paths of two cameras and eliminate undesirable vertical disparity as one of the main sources of visual fatigue;Disparity map estimation and 3D depth analysis to avoid depth budget/range exceeding and stereoscopic window violation.The research presented in this paper aimed to propose a complete image analysis method for live monitoring of the 3D video quality during professional recordings on a film set. The proposed algorithms, prototype diagnostic system, and software implementation were developed to ensure two main functionalities important during 3D video recording:Professional systems for recording 3D movies consist of two independent film cameras mounted on a dedicated mechanical structure called a stereoscopic rig. The rig allows changing the relative position of the cameras and, most importantly, setting the selected convergence. Due to their mechanical structure, rigs are divided in parallel  and perpendicularly (where the cameras are mounted perpendicularly using a semi-transparent mirror). Camera calibration aims to align the optical paths of both cameras to obtain a consistent and correct stereoscopic image. This procedure must always be performed at the beginning of the shooting day and repeated many times during the shooting. For this reason, it should not be a long process and at the same time precise, because the quality of the recorded image depends to a large extent on it. The proposed calibration method is based on the dedicated reference board with markers detectable by the algorithms implemented in the system (details can be found in ).The paper focuses on the second functionality: disparity map estimation for the needs of 3D depth monitoring on the movie set while recording the stereoscopic image. The analysis at the first stage of 3D video production aims to avoid visual fatigue sources such as stereoscopic window violation and vergence-accommodation conflicts. In general, the estimation and analysis of the disparity map can be performed in post-production. However, errors detected at this stage cannot be fixed in many cases or their repairing may entail high time or monetary costs. Therefore, the disparity map should be estimated and monitored on the movie set during stereoscopic video recording.The nature of the work on the film set  imposes specific requirements and expectations related to the auxiliary systems used during recording. They should improve and support the activities performed on the film set and not complicate them even further. Only in such a case their application is justified and brings noticeable benefits to the film crew.Therefore, the diagnostic system supporting disparity map estimation and analysis should be easy to use, which means requiring as little configuration activities and user interaction as possible. The system also should not require additional connections or signals to be provided mainly for its purposes. Ideally, it should produce valuable results only from signals that are already standardly available on a film set. As a diagnostic tool for the film crew, the system should generate a disparity map that faithfully reflects the cameras’ image without distortions and losses. The results, indicators, and meters produced by the system should be easy to unambiguously interpret. The system should offer capacity sufficient to process video streams typical of modern cinema (at least full HD resolution) on an ongoing basis (at least one frame per second).Based on the above requirements and discussions with the creators of 3D movies, two different system architectures were considered. One of them is based on the computer equipped with a Central Processing Unit (CPU), GPU, and extension cards for image acquisition. Another one assumes the development of a dedicated electronic device based on an Field-Programmable Gate Array (FPGA) circuit. Considering that the development of firmware for FPGA devices is complicated and time consuming, it was decided that the architecture of the system be based on the CPU and GPU. This solution is more flexible, easier to use, maintain, and extend, and gives more freedom in system development. The architecture of the developed system is presented in As already mentioned, the heart of the system is a computer equipped with a CPU and GPU accelerator. The film crew can select the form factor of the computer according to the needs and limitations of the film set. It can be a desktop computer, notebook, or industrial computer mounted in the rack together with other equipment. The selected computer needs to be extended with an image acquisition card(s) supporting SDI, which is a common standard used in the modern film industry. The system supports SDI input cards manufactured by Blackmagic Design company. This company offers a wide range of SDI-based solutions. Their extension cards provide from 1–8 SDI inputs and can communicate with the CPU using an Universal Serial Bus (USB) (solution preferred for notebooks) or Peripheral Component Interconnect Express (PCIe) interface . The system can handle more than one card. Its final configuration depends on the user.The SDI extension cards are connected to two cameras mounted in the stereoscopic rig. In general, professional cameras dedicated to cinematography store all data on removable disks attached directly to the camera. However, they are equipped with an additional SDI output for image preview. The preview image is distributed throughout the entire film set and available for all the devices such as monitors, recorders, etc. Therefore, the natural choice was to use these signals from both cameras as the inputs of the diagnostic system. These signals are easily available on the set, and their quality is enough for performed analyses.As the professional cinematography stereoscopic system comprises two independent cameras, an important issue is their precise synchronisation, correct acquisition, and the correlation of both video streams. As shown in As already mentioned, the computer used in the system should be equipped with a GPU card. Most of the modern GPU cards support general-purpose computing and can be used as a co-processor for the CPU. The General-Purpose Graphics Processing Unit (GPGPU) cards are very efficient in image processing. Therefore, some parts of the implemented algorithms use the GPU to obtain higher performance see .The presented architecture is relatively simple, and according to the requirements, the system is easy to build and maintain. For proper operation, it requires only two connections to camera preview outputs. As mentioned, these signals are distributed anyway throughout the film set . This approach makes the system non-invasive, significantly facilitates its use, and allows its installation far away from the rig.Determining a disparity map on a film set during the recording process allows the creators to analyse, evaluate, and if necessary, correct the quality of the 3D image. The generated disparity map should reflect the image observed by the camera as closely as possible, without distortions and clipping the image area. Therefore, the main objective of the work was to develop appropriate, efficient algorithms for image preparation (rectification) and determining a disparity map. The methods of stereoscopic image rectification can be divided into two main groups: calibrated and uncalibrated. The calibrated rectification methods, in addition to the input images, also require knowledge of the intrinsic  and extrinsic  camera parameters ,14. ThanThe uncalibrated rectification methods require only two input images, which is their main advantage from this system point of view. The use of these methods means that the system can only be connected to SDI signals from cameras and does not require any additional actions from the user . The uncalibrated methods are usually based on detecting and matching the image features. This means that they are flexible, but at the same time, more complicated, and hence computationally complex and less efficient.The review and evaluation of the available rectification algorithms showed that many are not suitable for application in a real-time diagnostic system. The authors evaluated rectification algorithms implemented in two popular image-processing frameworks—the Open Source Computer Vision (OpenCV) library and the Image Processing Toolbox for MATLAB. Both implementations are based on algorithms using a fundamental matrix to describe the epipolar geometry see ,16). Ano. Ano16])The research on uncalibrated rectification algorithms was conducted using 25 various example movies registered using a professional stereoscopic rig. All the above algorithms were discarded mainly because of distortions in the output image. The rectified images were often significantly distorted . Another reason was the unsatisfactory processing efficiency. The execution time for full HD test images ranged from a few to several seconds per frame.Speeded-Up Robust Features (SURF) algorithm [Features’ detection: Key points of the individual images are detected using the lgorithm  and corrRemoving outliers: Pairs of key points that have been assigned incorrectly in the previous step are removed from the list in order not to affect the calculations in the next steps. The deletion is based on the distance of the points in the pair and their X and Y coordinates;Target features’ calculation: Target key points are calculated in such a way that the horizontal coordinate is the same as for input image, while the vertical coordinate is the average value of the original key points;M × N uniform grid mesh, and every original key point is then mapped to the appropriate mesh cell according to its coordinates;Features-to-mesh mapping: The image is divided into an Bilinear inversion: The key points (both source and target) are represented as a linear combination of the four vertices of the mesh cell;Target mesh calculation: The target mesh cell calculation is defined as a linear least-squares problem that moves the key points to the target positions and minimises visual distortion. The implementation uses sparse linear system solvers available in the Eigen library ;Warping: This is the mapping pixels of the original input image to the rectified output image. The warping is guided by the mesh vertices’ position. To simplify the transformations, the forward warping method was selected for the implementation. The pixels of the output image that were not set during the mapping are calculated based on values of neighbouring pixels .The literature study and evaluation of the selected algorithms carried out using recordings from a professional stereoscopic rig showed that the most satisfactory results were obtained using the keystone correction algorithm proposed in . The algThe evaluation of the execution time of the above steps showed that the most time-consuming parts of the algorithm are warping and (for dense mesh) calculating the target mesh coordinates. Detailed measurements of the execution time of all individual steps are presented in https://www.techpowerup.com/gpu-specs/geforce-gt-650m.c547 (accessed on: 12 December 2021)) and NVIDIA GeForce GTX 780 Ti  (https://www.techpowerup.com/gpu-specs/geforce-gtx-780-ti.c2512 (accessed on 12 December 2021)). The parallelised implementation allowed achieving five-times acceleration for the first GPU card and twelve-times for the second GPU card. Detailed results are presented in The GPGPU version of the warping algorithm was implemented using Open Computing Language (OpenCL) to ensure portability to other operating systems and platforms. The main profit of the GPGPU implementation is to parallelise the processing of pixels into many independent threads. The size of the working group is parametrised so the application can be adapted to the hardware platform used. The algorithm code is divided into two kernels. The first is responsible for transforming pixels from an input image to an output image. It takes the input image pixels and calculates the new coordinates using target mesh vertices. Then, it copies the pixel from the input to output image and marks the output pixel coordinates in an additional mask image. The second kernel checks in the mask image which pixels in the output are not filled, and based on this information, it calculates their values using the data from neighbouring pixels. The GPU implementation was tested using two GPU cards: NVIDIA GeForce GT 650 M   that controls all the hardware components, ensures the acquisition and synchronisation of video streams, performs data processing, and visualises the raw data and analyses the results. The software application is the only system component with which the user interacts. This means the GUI should be ergonomic and intuitive to ensure easy operation in a very dynamic environment such as a movie set. On the other hand, the software also implements all the image-processing algorithms, which need to be very efficient and robust.Device drivers—responsible for communication with the hardware components. Device drivers provided by hardware manufacturers, together with related low-level libraries, support discovering all connected devices, communication and control, as well as the acquisition of raw data from the cameras;Image-processing libraries—provide all the algorithms required for camera calibration and disparity map estimation;GUI application—merges all lower-level components and ensures video acquisition, synchronisation, processing, and visualisation.All the components that make up the user application can be divided into three main layers:The application was developed for the Windows operating system and tested using Windows 7 and Windows 10. Still, one of the assumptions for future development is migration with macOS, which is commonly used in the cinema industry. Therefore, all the tools and libraries used in the developed software are supported by both operating systems. The primary development environment was the Qt toolkit. It is an open-source environment supporting the development of cross-platform applications and creating modern graphical user interfaces. The software also uses the OpenCV library for image-acquisition and -processing purposes. The library is an extensive collection of computer vision algorithms optimised for efficiency and real-time processing. Some image-processing algorithms implemented for the needs of the presented diagnostic system can be executed using a GPGPU. These algorithms were implemented using the OpenCL framework. This was selected because it ensures cross-platform compatibility. Thanks to OpenCL’s usage, the software can be executed across heterogeneous platforms, taking into account various operating systems and GPU types and architectures.Main Thread of the application is responsible for handling and monitoring all events happening in the system. First, it communicates with the libraries responsible for detecting and managing the SDI input modules. Based on the information provided by these libraries, two Frame Grabber objects are created, one per SDI input. The objects are wrappers providing the interface for the Qt and OpenCV libraries to configure the devices and acquire a full video stream.The Pre-processing Threads. The pre-processing includes, but is not limited to, image rotations and flipping, which are especially useful in the case of a mirror rig. After the pre-processing, both streams are transferred to the Synchronisation Thread. It creates stereo pair images based on the timecode information attached to each video frame. After synchronisation, both frames are considered inseparable stereo pair image, and as such, they are processed by all other application components.The acquired raw data streams from both cameras are transferred to Calibration Threads and Disparity Map Thread. The operation mode of the application can be changed during the operation by selecting the appropriate view. Threads related to a given operation mode are properly started and stopped so as not to overload the processor with unnecessary tasks. The Calibration Threads execute the algorithms aimed at the detection of the calibration reference board (separately on the left and right image) and the calculation of the camera alignment coefficients (using results from both images) [Disparity Map Thread is responsible for disparity map estimation. For this purpose, it first rectifies the images using the uncalibrated rectification algorithm and then calculates the disparity map (see The synchronised video stream can be used by the  images) . The Dis map see .An exemplary view of the main application window is shown in Storage, the number of invalid pixels in the disparity map calculated without rectification was 47.54%, and the average error in the calculated disparity was 4.48 px. Executing the rectification algorithm before calculating the disparity map reduced both indicators regardless of the mesh size. For example, for a mesh with five rows and five columns, the number of invalid (undetermined) pixels was reduced to 43.68 % and the average disparity error to 4.31 px. The results obtained for all the datasets showed that the implemented algorithm works correctly and, for meshes other than 1 × 1, positively influences the accuracy of the generated disparity map for almost all evaluated datasets. It should also be noted that increasing the mesh size improved the quality of the map.The algorithms for the rectification and determination of the disparity map implemented in the developed system were tested in terms of functionality and performance. The functional verification was carried out mainly based on the Middlebury Stereo Datasets . The aimThe rectification algorithm was also tested on sample movies recorded on a real film set. It was a set of 25 films shot under various conditions, both indoors and outdoors. The videos were recorded in full HD resolution (1920 × 1080 pixels). Each movie contained approximately 150 frames. A common problem in uncalibrated rectification methods based on detecting image features is the incorrect identification of key points. This results in a wrong rectification and hence the deterioration of the correlation between the left and right image. The tests aimed to check the quality of the rectification on the pictures from the film set, and the measure was the average vertical disparity calculated for a set of selected key points before and after the rectification. The evaluation consisted of determining several dozen key points using the SURF algorithm in the right and left images of the stereoscopic pair. The points were then paired, manually verified, and outliers discarded. For the rest of the points, the average vertical disparity was calculated. The underlined values show the cases when the average vertical disparity after rectification was smaller than before. The results showed that the rectification algorithm worked correctly for almost all test videos. Only in the case of one film, the mean vertical disparity was greater after rectification. It is important that even in the case of movies shot at a low convergence angle , the rectification algorithm did not cause erroneous image distortions and magnification of disparity.In addition to functional tests, performance tests of the full-image-processing path, including the rectification and calculation of the disparity map, were also carried out. The performance tests consisted of processing all 25 test films and measuring the execution time of individual steps of the algorithms. Two implementations of the algorithms were tested—one based only on the CPU and one also using the GPU. The tests were performed on two typical personal computers—desktop and notebook. The desktop computer was equipped with an Intel Core i7-4790 CPU, 32 GB of Random-Access Memory (RAM), and an NVIDIA GeForce GTX 780 Ti GPU card. The notebook was equipped with an Intel Core i7-3630QM CPU, 8 GB of RAM, and an NVIDIA GeForce GT 650 M GPU card. The results are presented in As can be seen, the use of the GPU allowed significantly reducing the time of warping and determining the disparity map. The resulting acceleration is shown in In the case of the notebook, the obtained acceleration was 4–5-times, and the average processing time required to calculate the disparity map was about 1 s per frame. In the case of the desktop computer with much better performance parameters, the speedup was about 10–12-times, and the total processing time was around 300 ms. As shown in The proposed diagnostic system was evaluated on film sets during the production of 3D films (in cooperation with the Polish companies FINN Sp. z o.o. and Isyrius). It was tested with two different camera models: EPIC MYSTERIUM-X manufactured by RED Digital Cinema and Phantom Flex4K from Vision Research. The system correctly acquired and synchronised the video streams for both of them. The developed method implemented in the prototype system gave results satisfactory for the film crew and allowed the detection of possible problems in the recorded image. It is possible to assess whether the recorded stereoscopic material is correct, does not violate the depth budget, and meets the assumptions for a given scene. An exemplary window with system processing results is shown in The main result of the research presented in this paper was the efficient GPU-based implementation of algorithms dedicated to the support of the film crew in analysing the quality of the recorded stereoscopic image. The proposed solution allows near real-time determining of the 3D disparity map in parallel with the image-recording process. The method was implemented and evaluated in the prototype diagnostic system dedicated to being used directly on the film set to acquire, synchronise, and process the video streams from stereoscopic converged rigs.The implemented algorithms were tested as part of the automated system developed for monitoring the 3D image quality on a film set. The main goal of the system is to provide a disparity map and histogram with the best possible reflection of the recorded image. The proposed approach and high-performance implementation of the algorithms using a GPU support allowed the development of the diagnostic system, which can generate and mark a disparity map on an ongoing basis. Its advantage is the lack of a need to calibrate and know the parameters of the cameras, thanks to which it easily adapts to frequent changes on the film set. The implementation of the selected algorithms using a GPGPU accelerator resulted in a 10-fold acceleration of the calculations and a reduction in the time for determining the disparity map to 300 ms. The processing efficiency at the level of a few frames per second (at full HD resolution) allows for relatively comfortable work with the system and tracking the image quality in parallel with its recording. Thanks to the uncalibrated image rectification algorithm, the system easily adapts to frequent changes of the camera configuration, which makes it easy to use and non-invasive in the entire filming process.The developed universal graphic software can be used on any hardware platform, enabling it to be used in various environments and film sets. However, the performance evaluation showed that it is still possible to optimise the processing algorithms, especially those related to image transformation and solving sparse matrix equation systems. It is also worth considering migrating and testing the developed algorithms on modern miniaturised GPUs (such as NVIDIA Xavier). The embedded GPU modules can further miniaturise the entire system and extend its scope of application.


Therefore, we propose a novel machine learning algorithm to assess the p-value of a feature interaction, named the extreme gradient boosting machine for feature interaction (XGB-FI). The first step incorporates the concept of statistical methodology by stratifying the original data into four subgroups according to the two interactive features. The second step builds four XGB machines with cross-validation techniques to avoid overfitting. The third step calculates a newly defined feature interaction ratio (FIR) for all possible combinations of predictors. Finally, we calculate the empirical p-value according to the FIR distribution. Computer simulation studies compared the XGB-FI with the multiple regression model with an interaction term. The results showed that the type I error of XGB-FI is valid under the nominal level of 0.05 when there is no interaction effect. The power of XGB-FI is consistently higher than the multiple regression model in all scenarios we examined. In conclusion, the new machine learning algorithm outperforms the conventional statistical model when searching for an interaction.Recent studies have revealed the importance of the interaction effect in cardiac research. An analysis would lead to an erroneous conclusion when the approach failed to tackle a significant interaction. Regression models deal with interaction by adding the product of the two interactive variables. Thus, statistical methods could evaluate the significance and contribution of the interaction term. However, machine learning strategies could not provide the  Another study [Recent studies of the interaction effect in cardiac research successfully discovered crucial findings. Electromagnetic interactions between implanted cardioverter defibrillators and left ventricular assist devices were examined . An inteer study  estimatep-value is less than the significance level of Regression analysis detects interaction between two independent variables by the product of the two independent variables Compared with conventional statistical approaches, machine learning algorithms improve the predictive accuracy by fitting the model with a tremendous training process . Driven Among the learning algorithms, there are unsupervised and supervised learning. Unsupervised machine learning only examines the associations between a set of predictors , for whiIn addition to the SVM, tree-based designs are the most popular due to simple applications and interpretations. Random forest could provide feature importance through decision trees . The algInstead of using the bagging technique in random forest models, the gradient boosted decision tree (GBDT)  builds oRecent research  suggestSince the tree-based design has the most intuitive structure to deal with interactions, this research aims to develop an algorithm to evaluate the statistical significance of a feature interaction by modifying the data structure of the XGB machine. The new approach is named the extreme gradient boosting machine for feature interaction (XGB-FI).Build the XGB machine using the original dataset without the two interactive features  for all possible combinations of predictors as p-value using the threshold Q1 is the first quartile (25th percentile), and IQR is the interquartile range, according to the empirical FIR distribution.Declare significance or obtain the empirical The training and testing sets contain 80% and 20% of the data, respectively. Assuming there is only a single two-way interaction, the XGB-FI algorithm has five steps with 10-fold cross-validation to avoid overfitting:It is worth noting that the above algorithm adopts the same hyperparameter of the XGB by grid search using the whole data with every feature. The learning rate is 0.05. The number of trees is 300. The early stopping round is 10. The maximum depth of a tree is 6. The rest hyperparameters are default settings.In the first step, the XGB machine is built using the original dataset without the two interactive features. We denoted the model as In the second step, a concept of stratification is adopted that eliminates the impact of interactions . AssuminNote: X1 and X2.). We denoted this model as In the third step, we implement another four XGB machines within each stratum and then derive the mean of the four RMSEs. It is worth noting that the four subsets do not contain the two variables with feature interaction  asX1 and X2. In contrast, the numerator is correctly adjusted for feature interaction but not the main effect. As a result, the difference between the numerator and the denominator is only the interaction effect.The denominator of the FIR represents the RMSE not controlling for interaction and the main effect of K, there are As the numerator represents the gain of treating the interaction impact, the smaller FIR indicates a more substantial interaction effect. This step calculates numerous FIRs for all possible combinations of interactions. If the total number of predictors is Q1 is the first quartile (25th percentile), and IQR is the interquartile range, according to the empirical FIR distribution of p-value for the feature interaction. However, we discovered that the 95th percentile obtained from the empirical distribution did not provide the correct significance level. Therefore, we used a nonparametric threshold for outliers in the box plot.The last step declare statistical significance with the threshold: . We also explicitly indicated the threshold to declare significance based on the empirical distribution of all null interactions in the set of predictors. Computer simulations confirm that the XGB-FI has a valid type-I error rate under the null hypothesis of no interaction effect. Most importantly, the XGB-FI outperforms the conventional regression model in detecting the interaction effect.The limitation of XGB-FI is that at least 15 predictors are required to generate the null distribution of FIR for all possible interactions. If the data contain less than 15 features, then the combination of potential interactions is less than 105, and the probability of declaring significance drops accordingly. More predictors would result in a more reliable null distribution of the FIRs and assure the statistical power of the XGB-FI. Although the simulations assume continuous features, scenarios for categorical or ordinal feature interactions are intuitive.p-value under a higher-order interaction effect.This research focused on a two-way feature interaction. However, a higher-order interaction such as a three-way interaction could adopt similar concepts. Future studies could carry out the empirical distribution of the FIRs and declare significance or assess the Similar to a previous study , simulatIn computer simulations, we adopted a similar approach in our recent research  and geneAlthough this study used simulated data, the XGB-FI could be implemented in cardiac research or other fields. The discovery of more complicated interaction effects will benefit tremendous clinical applications.The limitation of this research is that we only simulated continuous predictors and outcomes. However, feature interaction is a very complex and nuanced area of study. There are other combinations with nominal and ordinal variables that introduce feature interaction, the nature of the interaction, the localization of the interaction effect, or the range of effect. Therefore, the XGB-FI may not be the optimal strategy without considerable research, and a future study is highly desired.


Pneumatic conveying is a standard transportation technique for bulk materials in various industrial fields. Flow metering is crucial for the efficient and reliable operation of such systems and for process control. Capacitive measurement systems are often proposed for this application. In this method, electrodes are placed on the conveyor systems transport line and capacitive signals are sensed. The design of the sensor with regard to the arrangement and the number of electrodes as well as the evaluation of the capacitive sensor signals can be divided into two categories. Calibration-based flow meters use regression methods for signal processing, which are parametrized from calibration measurements on test rigs. Their performance is limited by the extend of the calibration measurements. Electrical capacitance tomography based flow meters use model-based signal processing techniques to obtain estimates about the spatial material distribution within the sensor. In contrast to their calibration-based counterparts, this approach requires more effort with respect to modeling and instrumentation, as typically a larger number of measurement signals has to be acquired. In this work we present a comparative analysis of the two approaches, which is based on measurement experiments and a holistic system model for flow metering. For the model-based analysis Monte Carlo simulations are conducted, where randomly generated pneumatic conveying flow patterns are simulated to analyze the sensor and algorithm behavior. The results demonstrate the potential benefit of electrical capacitance tomography based flow meters over a calibration-based instrument design. The transportation of bulk materials by means of pneumatic conveying is widespread in a variety of industrial fields. Examples are steel making, power generation, agriculture and food industries as well as pharmaceutical and chemical industries ,2. A pneHowever, flow measurement of pneumatic conveying flows is a difficult measurement problem, in particular within horizontal pipes. Different flow regimes occur, which refers to the spatial and temporal distributions of the particles within the transport process and the corresponding velocity profiles. A current and comprehensive classification about several flow conditions, which can occur in pneumatic conveying can be found in . A summaThe mass concentration as well as the particle velocity of horizontal pneumatic flows are in general inhomogeneous and show a spatial dependency. Hence, the mass flow rate lated by :(1)m˙t=∫x and y are Cartesian coordinates and t is the time. Due to the spatial dependency of the quantities of interest, mass flow metering in pneumatic conveying is in general a challenging task.Hereby, Capacitive sensing technology has been suggested and used for flow measurement in pneumatic conveying ,9,10. Fiv is an average velocity. Information about the particle velocity can be obtained from the time series of the measurement signals. Hereby, frequency analysis of the capacitive signals from dedicated electrode structures [The first approach is based on an empirical determination of the relationship between the capacitive measurement data and the average mass concentration unctions ,12. The ructures ,14 or coructures . The calructures ,16 for tThe second approach is based on model-based signal processing techniques from the field of electrical capacitance tomography (ECT). Hereby, the spatial dielectric material distribution within the sensor is estimated from the capacitive measurements ,18. The The different nature of the approaches raises fundamental technical questions, e.g.,How does the number of electrodes influence the performance of the flow meter, orWhat is the potential benefit of the ECT-based approach with respect to the calibration-based approach.The list of questions is not yet exhausted and could easily be extended to include other technical aspects like the signal quality etc. Yet, the presented two questions should be addressed first in order to point out the potential advantages of the two approaches.Due to the variety of flow regimes, which can occur in horizontally aligned pneumatic conveying systems see  a metrolIn , the autIn this paper, we present an analysis of the two flow measurement approaches using the proposed model. Hereby, we consider the determination of the mean mass concentration The main contributions of the results and approaches presented in this article therefore areAn analysis of the influence of the number of electrodes of the sensor,An analysis of different signal processing methods for capacitive flow metering andReference measurement procedure to parametrize/validate the model for specific sensor evaluations.Thus, the paper contributes to the research questions raised for the two instruments types. The demonstration of the holistic model for this analysis provides a novelty for this field, as it allows a comprehensive analysis for the whole range of flow regimes.This paper is structured as follows. In this section, the holistic modeling of the measurement chain of a capacitive mass concentration measurement system for pneumatically conveyed solids is discussed. The scheme of the simulation model is depicted in The aim of the statistical process model is to generate random samples of mass concentration distributions as they occur within pneumatic conveying processes. As shown in A homogeneous mass concentration over the whole cross-section of the pipe corresponding to the dispersed and slug flow regimes andA dense lower phase with a certain height and a dispersed upper phase corresponding to flow regimes with a distinct material layer at the bottom of the pipe. Hereby, the mass concentration of the lower phase is not necessarily the bulk density of the material since the gas stream can aerate the transport good .N is the number of FEs used for the discretization of the inner region of the pipe. Exemplary random samples of PP pellets with a bulk density of For the first case random samples are generated by applying a constant random mass concentration  applied . TherefoThe purpose of the material model is to describe the relationship between the mass concentration and the relative permittivity of aerated bulk materials, as they occur within pneumatic conveying processes. A suitable model, which describes this relationship is the Landau–Lishitz–Looyenga (LLL) Equation ,20. For Hereby,  m−3. In , a wide In , a dedicEquation  is simplfound in ,31.Applying the model Equation  to the iGiven the dielectric material distribution represented by the vector Before the measurements are provided to the estimator for ort good ,32. In tThe last element of the holistic model is the estimation of For the data evaluation in calibration-based capacitive flow meters it is common to select a suitable function set for test rig ,12. E.g.For the ECT-based flow meter approach image reconstruction algorithms are applied to estimate the spatial dielectric material distribution and subsequently found in ,24. For found in . DetailsIn this section, the measurement setup for the parametrization of the holistic simulation model is presented. The comparative analysis of the two different instrument approaches are later based on this setup. Furthermore, reference measurements for the validation of the system model are discussed.The sensors are connected to the front-end circuitry by means of coaxial cables. The front-end circuitry is given by a displacement current based low-Z measurement system ,33,34. HIn order to analyze different sensor designs and also to validate the results by means of measurement experiments we propose the emulation of sensors with different numbers of electrodes by combining the measurements of the existing sensor assembly with eight electrodes. This procedure is schematically depicted in h of PP granulate are filled within the material holder to produce horizontal material layers comparable to distinct bottom layers of horizontal flow patterns  of the differences between the average trend of the simulated estimates After the simulation model has been validated, it is used for a comprehensive analysis of different sensor designs and signal processing algorithms. This study is carried out by means of simulations. Hereby, random samples of the flow conditions are generated for all possible flow regimes, which can occur in pneumatic conveying systems. For this purpose, the statistical process model described in ated see , case 2.ated see  is evaluIn this subsection the analysis of the influence of different numbers of electrodes is demonstrated. First, the measurement results for model validation are shown. Then, the simulation-based uncertainty analysis for the flow regimes of pneumatic conveying is performed.e over the range of The even PP layers used for the measurement experiments exhibit sharp transitions between the lower and the upper phase resulting in relatively large estimation errors. This can be attributed to the soft field nature of ECT, which causes a blurring of sharp boundaries . In actuy-axis (see y-axis (see axis see . Since taxis see  an unsymIn this subsection the influence of different ECT signal processing techniques is studied. The analysis is carried out for an ECT-sensor with eight electrodes.e stated in Equation (The results depicted in Equation  over theFor a deeper analysis of capacitive flow metering in pneumatic conveying, the validated holistic simulation model is used to evaluate the different ECT-based estimation algorithms for the average cross-sectional mass concentration for all possible flow regimes, which can occur in pneumatic conveying systems.The results presented in this section clearly demonstrate the potential benefit of ECT-based flow metering maintaining a larger number of electrodes, with respect to the calibration-based approach.Due to the properties of the flow patterns, different material distributions the same cross-sectional average mass concentrations can result in varying capacitances of a single electrode pair. Hence, in addition to measurement noise, the conveying process itself causes an intrinsic uncertainty of the estimates . IncreasFurthermore, the benefit of improved modeling and signal processing techniques for ECT has been demonstrated. Although the results are in favour of ECT, the development of a flow meter has also to consider aspects like the instrumentation and hardware effort to conclusively find a suitable solution for a specific flow measurement application. This aspect was yet not considered in this work. However, the presented approach can be used for such considerations by adapting the model for the specific application.The analyses have been carried out for the average cross-sectional mass concentration  systems ,37.In this paper, fundamental properties about a calibration-based and an ECT-based approach for capacitive flow metering in pneumatic conveying have been analyzed with respect to the determination of the mass concentration of the transport good. The study is based on a holistic simulation model of the measurement process and validation measurements on a test rig. In particular the influence of the number of electrodes and different estimation approaches for the determination of the mass concentration of particle/gas mixtures were analyzed. The results show the potential benefit of ECT-based flow metering over a calibration-based instrument design. It was demonstrated how the RMSE of the estimates can be decreased for sensor designs with increased number of electrodes and how further improvements are possible, when refining application tailored estimation algorithms. The model based analysis approach can further be used to address relevant development questions for flow metering in pneumatic conveying.