BrainK is a set of automated procedures for characterizing the tissues of the human head from MRI, CT, and photogrammetry images. The tissue segmentation and cortical surface extraction support the primary goal of modeling the propagation of electrical currents through head tissues with a finite difference model (FDM) or finite element model (FEM) created from the BrainK geometries. The electrical head model is necessary for accurate source localization of dense array electroencephalographic (dEEG) measures from head surface electrodes. It is also necessary for accurate targeting of cerebral structures with transcranial current injection from those surface electrodes. BrainK must achieve five major tasks: image segmentation, registration of the MRI, CT, and sensor photogrammetry images, cortical surface reconstruction, dipole tessellation of the cortical surface, and Talairach transformation. We describe the approach to each task, and we compare the accuracies for the key tasks of tissue segmentation and cortical surface extraction in relation to existing research tools . BrainK achieves good accuracy with minimal or no user intervention, it deals well with poor quality MR images and tissue abnormalities, and it provides improved computational efficiency over existing research packages. With the simple recording of the electroencephalogram (EEG), the brain's electrical activity can be measured with millisecond temporal resolution at the head surface. Dense array EEG (dEEG) systems now allow up to 256 channels to be applied quickly with full coverage of the head, assessing the fields from the basal as well as superior cortical surface , 2. The To model cortical sources with these properties, accurate cortical surface extraction is a key challenge. Furthermore, the volume conduction of the electrical potentials, from the cortex to the head surface, must be specified through characterizing the conductivity of each tissue compartment. The skull is the primary resistive medium in the head, and it must be modeled, preferably with bone density values from CT. Because the electrical boundary effects of the volume conduction are affected by discontinuities in current paths, for example, caused by holes in the skull , spherical shell or boundary element models provide only approximate electrical propagation from cortex to the surface, and more detailed (FDM or FEM) volumetric models are needed. Finally, the position of the electrodes must be specified accurately, for example, with geodesic photogrammetry .By describing the positions of electrodes and cortical surface targets, an accurate electrical head model also supports dense array approaches to transcranial current injection. The electrical head model can be validated with bounded Electrical Impedance Tomography (bEIT), in which current injection and potential recovery are analyzed, within the bounds of the tissue geometry from MRI, to test whether the electrical conductivity of head tissues estimated by the model is accurate in predicting the recovered potentials , 6. WithIn the present report, we review the methods implemented in BrainK. In addition, we report validation studies for accuracy and efficiency of both tissue segmentation and cortical surface extraction. Klauschen et al.  evaluate conformal Atlas head model. If only a structural MRI  is available for the person, the sensor positions are registered to the MRI, a database skull (from CT) is registered to the MRI, and tissue segmentation and cortical surface extraction are conducted to create the individual head model with Atlas skull. The skull compartment includes either the original CT Hounsfield units for estimating bone density (and thus conductivity) voxel by voxel or a single skull segment, depending on the FDM computational model that will use BrainK's head model output. If both the CT and MRI are available for the person, the MRI is registered to the CT, which has more accurate dimensions than the MRI, to create the individual head model with individual CT skull.For all workflows when the T1 MRI is available for the person, BrainK segmentation identifies the white matter (WM) and the gray matter (GM). It partitions these into two hemispheres and it differentiates cerebellum from cerebrum. In addition, an entire head mask and the two eyeballs are separated as well in the segmentation component. The eyeballs may be important for the electrical head model because of the large far fields generated by their cornea-retinal potentials . For all workflow scenarios, a generic spherical sensor cloud is warped onto the head contour of the subject. For the scenarios of Atlas-to-MRI, MRI-to-CT, and CT-to-MRI, an additional GPS-to-head registration procedure is conducted to register the specific GPS sensor cloud, from one individual Geodesic Sensor Net application, onto the head contour. For the scenarios of Atlas-to-MRI-to-GPS and Atlas-to-GPS, the individual GPS data has already played a role in the skull registration and has been aligned with the head contour. In all scenarios, the skull registration is conducted, such that the resulting head segmentation includes the following tissue types: WM, GM, CSF, bone, flesh, and eyeball, in which the WM and the GM are further partitioned into two cerebral hemispheres and the cerebellum.The MRI segmentation implements a cascade of automatic segmentation procedures in order to identify and separate the head tissues required for electrical head modeling, including the scalp, skull, cerebrospinal fluid (CSF), brain (gray and white matter), and the eyeballs. Finally, cortical surface extraction is performed to allow characterization of the normal of the cortical surface . The cascade mainly consists of two types of procedures: voxel classification on a region of interest (ROI) and the extraction of certain anatomical structures. The classification procedure aims at labeling all voxels in the ROI into different tissue types. The extraction procedure performs morphological operations on the voxel classification so that a certain anatomical structure, such as the WM, the GM, or the scalp, is separated from erroneous  segmentation results.R1, and classifies all the voxels into two types: foreground and background. The following foreground extraction procedure does simple morphological operations to further improve the foreground segmentation resulting in the second ROI, R2, in which all voxels are classified into three tissue types: WM, GM, and CSF, as shown in R2, the WM extraction procedure then produces an initial WM volume W1, which is used as the basis to identify the initial GM volume G1 by the GM extraction procedure. Next, the union of G1 and W1 is taken as the third ROI, R3, in which the WM/GM classification is refined with a different RT scheme. The new WM volume is then processed by the WM extraction procedure and a WM partition procedure, ending up with the second WM volume W2.Given the MRI data, the segmentation cascade as shown in W2: two types of cerebral WM for the two hemispheres, respectively, and the cerebellar WM. W2 then forms the basis for extracting the new GM volume G2, which includes the two types of cerebral GM, respectively, for the two hemispheres and the cerebellar GM, as shown in W2 plus G2) is then taken as a reference data for the scalp segmentation resulting in the head mask Hm. The scalp segmentation then conducts a binary thresholding procedure on R1.This WM partition procedure not only separates the two cerebral hemispheres from each other but also makes an optimal cut between the cerebral WM and the cerebellar WM with the well-known maximum flow algorithm . The resTo achieve the above segmentation steps, BrainK implements a novel morphological image analysis (SMIA) technique  and a cer = 0.83 with water content in trabecular skull [Although segmentation of the skull from the MRI is an important challenge, the characterization of bone properties is poor in the typical T1 MRI sequence compared to that provided by CT. The Hounsfield units provided by the CT image (measuring X-ray attenuation) are directly proportional to bone density and thus provide important information on the relative tissue properties of trabecular skull. For example, the CT Hounsfield units correlate ar skull .The CT image segmentation procedure first performs the simple yet robust thresholding  so that A well-known and prominent image artifact that challenges MR image segmentation is the intensity inhomogeneity (IIH) due to spatial distortions in the radio frequency (RF) gain in the RF coil . The preThe nature of RT makes it robust against the influence of IIT without introducing an explicit IIH correction. Like traditional thresholding methods, RT enables exhaustive searching without possibilities of being trapped into local optima. It also incorporates various a priori knowledge such that it is robust to intersubject variability and to the convolution of cortical structures. Finally, RT conducts brain tissue segmentation without the need of a prior step of brain volume extraction.bi and an additive noise field ρi as follows: yi is the observed intensity at the ith voxel and yi′ is the unobservable true intensity without the influences of IIH and noise.In spite of intersubject variability, complexity of cortical structures, and variability on MR imaging sequences, we can make the following a priori structural, geometrical, and radiological observations: (1) Skull, CSF, GM, and WM form a layered structure from outside to inside; (2) the average intensities of skull, CSF, GM, and WM in local regions are in ascending order in T1-weighted MRI; (3) the cortex thickness is nearly uniform and is a very small value compared to the size of brains; in addition to the structural modeling described above, we also formulate an image model by incorporating a multiplicative low-frequency bias field xi with that of a reference voxel xj on its gradient path, which is a set of ordered voxels emanating from xi and following the gradient direction at each voxel in the path. The first procedure, GM-WM differentiation, scans all voxels in the ROI, which are initialized as WM, and labels them as GM if there is such a reference voxel xj that its distance from xi is less than a distance threshold dgw and the ratio between their smoothed intensities zi/zj is less than a relative threshold tgw; the second procedure, CSF-GM differentiation, scans each GM voxel xi and relabels it as CSF if there is such a WM voxel xk on its gradient path whose distance from xi is less than another distance threshold dcw  (dcw > dgw) and the ratio between their smoothed intensities zi/zk is less than another relative threshold tcw  (tcw < tgw).The structural modeling and the image modeling enable two procedures, both of which compare the intensity of a subject voxel h = wwhw + wghg + wchc. In this equation, hw is a metric measuring the homogeneity of the WM object and formulated as the sum of smoothed intensity differences of any pair of adjacent voxels labeled as WM. hg and hc are formulated in the same spirit for GM and CSF, respectively. ww, wg, and wc are three weighting coefficients.In order to find the optimal relative thresholds, BrainK exhaustively tries each pair among the Cartesian product of the candidate GM/WM relative threshold set and the candidate CSF/WM relative threshold set and chooses the pair that minimizes an objective function R3 is first reset and then an initial set W0 of WM voxels are determined by thresholding R3 with a traditional threshold that maximizes the objective function f = fe − fr, where fe is the sum of intensity differences of all m pairs of adjacent voxels labeled as WM and non-WM and fr is the sum of differences of the m pairs of adjacent voxels labeled as WM and with the greatest intensity differences. A minimal cardinality of W0 is set to improve the robustness. The remaining region of R3 is then segmented as a procedure of iteratively dilating W0 by using the optimal GM/WM relative threshold to compare the subject voxel's intensity to a reference intensity computed by considering the WM voxels near the subject voxel.The second RT scheme takes the optimal relative thresholds and differs from the first scheme on the question of how to determine the reference intensity compared to the intensity of the subject voxel. The ROI W2, the GM volume is first created by a gradient flow process, in which any voxel originally labeled as GM in the ROI is taken as the true GM if it can reach any WM voxels by following the gradient path emanating from itself within a given distance threshold. Each step in the gradient path is determined by looking at the gradient direction for each voxel. This initial GM volume is then further processed by traditional morphological operations to improve the segmentation, ending up with the GM volume G2.The GM and WM extraction procedures are responsible for extracting the GM and the WM structure from the raw voxel classification in the ROI. We describe GM extraction first, assuming that WM extraction is already done. The GM extraction utilizes the following a priori knowledge: (1) the thickness of cortex is nearly uniform; (2) GM wraps around WM such that two tissues form a layered structure; (3) the average gray level of GM is lower than that of WM in any local region. Given the extracted WM volume WM extraction is a more challenging process than GM extraction because it is responsible for extraction of WM itself, and it is also responsible for the brain volume extraction. The following a priori knowledge forms the basis of WM extraction: the WM is highly connected, but it has no topological defects. Topological defects or false positives present narrow bottlenecks. The extraction mainly consists of two steps: (1) WM localization, in which the center of one of the hemispheres is determined; (2) separation of WM from false positives. The localization of the WM center is essentially finding the WM voxel with the highest connectivity, which means that some quantitative measurement of connectivity should be taken for all WM voxels. Since topological defects, such as a tunnel inside the WM, can greatly influence the geometric measurement of the connectivity, the WM should be first topology-corrected before the connectivity measurement. Because the cortex surface (ribbon) must be defined in relation to the correct topology of the WM, WM topology correction is also essential for valid cortical surface reconstruction. BrainK introduces a skeletonization based morphological image analysis (SMIA) method and a cell complex based morphometric image analysis (CCMIA) framework. The following sections describe how these methods play central roles in WM extraction through achieving accurate connectivity measurement and topology correction. surface skeletonization procedure and a curve skeletonization procedure, both of which are based on its extensive topological point classification. Given a binary image, we say its foreground F has a handle whenever there is a closed path in F that cannot be deformed through connected deformations in F to a single point. A handle in F is referred to as a tunnel in its complement F is simple if it can be added to or removed from F without changing the topology of both F and  simple point[ thick-simple point. A point P is thick-simple with respect to F if it is simple with respect to F and the removal of P and any of its neighbors from F does not increase the number of tunnels and number of connected components in F. A point is thin-simple if it is simple but not thick-simple and can be classified as thick-surface and thick-curve points. Thick-simple points can also be further classified as surface-edge, curve-end, and other point types.BrainK's SMIA framework consists of aple point is a cen thick-surface skeletonization and thin-surface skeletonization. The former results in a discrete surface with thickness of at most two voxels, and the latter results in a final thin surface skeleton of one-voxel thickness. The thick-surface skeletonization iteratively removes boundary points of the object that are thick-simple at the moment of removal. The thin-surface skeletonization iteratively removes boundary points of the thick-surface skeleton that are thick-surface points, thick-surface-edge points, or thick curve-end points at the moment of removal. The curve skeletonization procedure iteratively removes the boundary points of the object that are thick-curve points or thick-simple, but not curve-end points, at the moment of removal. As a side product, BrainK's surface skeletonization procedure gives each skeleton point a depth metric as the distance from the point to the boundary, whereas the curve skeleton procedure gives each skeleton point a wideness metric.There are two steps for BrainK's surface skeletonization based on the above point classification:If the curve skeletonization procedure is conducted at a certain scale, then the points that become the thin-curve points can be checked if they are on a handle of the object. Removal of such handle points then corrects the topology defects by cutting the handle. BrainK supports a multiscale topology correction  method o cell complex [X in the 3D binary image, we can construct a 3-dimensional cell complex, that is, 3-complex, by creating a point for each voxel, an edge for every two connected points, a triangle for every three interconnected edges, and a tetrahedron for every four interconnected triangles. CCMIA proceeds as transforming the 3-complex to a 2-complex with points, segments, and polygons only and then the 2-complex to a 1-complex with points and segments only. The 3-2 transformation iteratively removes those boundary tetrahedrons by dropping one face f while a connectivity metric and a depth metric are accumulated on each f′ of the remaining three faces for the removed tetrahedron as depth(f′) = depth(f) + d, while d  represents the distance between the centers of the two faces. In a similar spirit, the 2-complex can be transformed to 1-complex and the remaining edges can be set with two accumulated metrics as wideness(e′) = max(widness(e′), wideness(e) + d) and connectivity(e′) = connectivity(e′) + connectivity(e) + d.CCMIA is motivated by the goal of representing the true connectivity of an object's skeleton considering the convolution of such objects as the cerebral WM, where neither the depth nor the wideness of the structure is good enough for this need. It is essentially a series of transformations on a space called the complex . A cell After the raw WM object is topologically corrected, its surface skeleton is processed by CCMIA ending up with the 1-complex with points and edges only. The point incident to the edge of the greatest connectivity is identified as the WM center and a wideness threshold is used to break the bottleneck between the true WM and the false positives. Finally, the WM surface skeleton is dilated to restore the volumetric object.Given the varying data available, flexible and accurate registration is an essential BrainK function. Whereas the MR image accurately differentiates soft tissues, the CT image accurately represents bone. When individual CT or even MRI is not available, BrainK provides registration of the sensor photogrammetry data to a digital head tissue Atlas of the appropriate age and gender. BrainK integrates both soft tissues and bones from multimodal images and from existing digital Atlases by various image registration techniques.Image registration is a problem of finding optimal geometric transformations between images so that each point of one image can be mapped to a corresponding point of another image. There are generally rigid transformations and nonrigid ones. Rigid transformation involves only translation and rotation. Affine transformation is a typical nonrigid transformation and allows scaling and shearing. Another form of nonrigid transformation is an elastic transformation, allowing local deformation based on models from elasticity theory. BrainK implements rigid and affine transformations, as well as a landmark-based elastic transformation using thin-plate spline theory . For difBoth the rigid transformation and the affine transformation can be represented by a simple 3 × 4 matrix. The key issue in applying such transformations is to optimize the similarity measure between the source and the target image, which in BrainK is represented in terms of landmarks. For the thin-plate spline transformation, the displacement field at any point is computed by interpolation from the vectors defined by a set of source landmarks and a set of target landmarks. It can be seen that landmark extraction plays a central role in all three transformations. BrainK is optimized for extracting landmarks based on the preceding image segmentation and on a priori anatomical knowledge. All registration procedures in BrainK undergo three steps: (1) landmarks extraction, (2) transformation coefficients determination, and (3) transformation execution. Note that three transformations all use landmarks but in different ways. The landmarks in rigid and affine transformation are used for constructing the similarity measurement, while there are source landmarks and target landmarks in thin-plate spine transformations and they are mapped to build the displacement field.n source landmarks Pi =  in the source image and the corresponding target landmarks Pi′ =  in the target image, the thin-plate spline transformation maps any point P =  in the source image to the point P′ =  in the target image as follows:ri2 = (x − xi)2 + (y − yi)2 + (z − zi)2. The coefficients in the above equations together form a (n + 4) × 3 matrix W, which can be obtained by solving the equation LW = M that forces the matching of Pi and Pi′, where M is formed by organizing the coordinates of Pi′ =  and L is formed by composing the coordinates Pi =  and the distance information ri2lnri2.The thin-plate spline theory is based on an analogy to the approximate shape of thin metal plates deflected by normal forces at discrete points. Given When the individual CT as well as the MRI data is available for the person, the registration component in the MRI-to-CT scenario first segments the CT into the following tissue types: bone, brain, flesh, and background. The MRI segmentation is then aligned with the CT segmentation so that the brain tissues and the eyeballs recognized in the MRI data are properly transformed and put into the brain region defined by the CT data. The result of this transformation is that the entire head segmentation  is now registered with the CT volume, where the CSF is redefined as the “brain” region recognized in the CT segmentation, but now subtracting the WM and GM that were inserted from the MRI segmentation.When both MRI and CT are available, the user could also choose the CT-to-MRI registration so that the bone recognized in the CT is transformed and put into the MRI segmentation wrapping around the brain and the CSF is inserted between the brain and the cranium. However, the MRI-to-CT registration is preferred when the CT covers the entire head of the subject because MRI imaging introduces more geometry distortion than CT. However, when the CT only covers the cranium region and the MRI data includes the face and the jaw, the user could consider the CT-to-MRI registration in order to have adequate face and jaw data for the electrical head model. flesh.When only the MRI data is available for the given subject, BrainK supports the skull Atlas-to-MRI registration, in which the bone tissue from an Atlas dataset is warped into the MRI space and the CSF is also taken as those regions between the cranium and the brain. Any region within the head mask that is not occupied by the GM, the WM, the CSF, the eyeballs, or the bone is set toWhen only the MRI and the individual GPS data are available and if the MRI is badly warped , the user could choose the MRI-to-GPS registration, so that the head region is warped to match the GPS sensor cloud, which has good metric properties of the actual head shape. The result is transformed segmentation data, which can be further processed by the Atlas-to-MRI registration described above.Thus, the skull registration can be adapted to fit the available image data for the individual. When only the individual GPS data is available for a given subject, an Atlas dataset with all required tissue types can be warped to match the GPS sensor cloud with the so-called Atlas-to-GPS registration, producing an individual conformal Atlas that has the appropriate shape of the volume conductor for electrical head modeling.Bm, which has some inner CSF space filled and the brain contour smoothed but still misses some CSF voxels mainly wrapping around the superior contour of the brain. MRI segmentation also produces the MRI head mask Hm and the two eyeball masks E1 and E2. Correspondingly, we have the CT brain mask Bc and the CT head mask Hc. These data form the input set used to determine the transformation coefficients of the MRI/CT registration.After MRI segmentation, we already have the MRI brain volume that is composed of WM and GM, but not CSF. The CT brain volume occupies spaces of WM, GM, and CSF. By applying morphological closing at a proper scale on the union of the MRI WM and GM, we obtain the MRI brain mask Bm. The similarity measure to optimize the rigid transformation is the sum of squares of the distances from all landmarks to the contour of Bc. The optimal coefficients are found by multiscale exhaustive searching with acceptable computational efficiency and good global optimization performance.MRI/CT registration undergoes a rigid transformation followed by an affine transformation. Different sets of landmarks are extracted for the two transformations. The rigid transformation aims at only providing a good initial state for the affine transformation and its landmarks are simply those bordering voxels around Bm while the superior landmarks are from the superior contour of Hm. The inferior and the superior landmarks are partitioned so that they “visually” wrap up Bm but do not overlap. The motivation of using two sets of landmarks is to align MRI with CT in terms of both the brain matching and the scalp matching. However, the brain volume in the CT may contain significantly more CSF than that in the MRI at the superior part of the volume and we make the observation that their contours have reliable matching only at the inferior part of the brain. The landmark extraction thus essentially involves the partitioning of the cephalic space into the inferior part and the superior part. BrainK takes the following three landmarks as reference points to partition the cephalic space: the center of the brain mask Bm and the centers of the two eyeball masks E1 and E2. All three points can be automatically determined.The landmarks for the affine transformation are more sophisticated and consist of two parts: the inferior landmarks and the superior landmarks. The inferior landmarks are from the border voxels of the inferior part of Bc and the other is the sum of squares of the distances from all superior landmarks to the contour of Hc. A gradient-descent optimization method is used to obtain the optimal affine coefficients. When all the optimal coefficients of both the rigid and the affine transformation are obtained, the MRI/CT registration can be performed from one direction while the other can be achieved by simply inverting the coefficient matrix. The result of MRI-to-CT registration is shown in The similarity measure for the affine transformation is formulated as the sum of two terms. One is the sum of squares of the distances from all inferior landmarks to the contour of Since the Atlas and the individual subject can have significant geometric differences, the Atlas-to-MRI registration undergoes an elastic thin-plate spline transformation following the rigid and the affine transformations. The rigid transformation provides a good initial state for the affine transformation, which in turn provides a good initial state for the elastic local transformation. The result of the Atlas-to-MRI registration is illustrated in Bm be the MRI brain mask defined as in MRI/CT registration. Let Ba be the Atlas brain mask defined in the same way as Bm. The landmarks are then the set of border voxels of Bm and the similarity measure is the sum of squares of distances from all landmarks to the contour of Ba. The coefficients of the rigid transformation are found by multiscale exhaustive searching while those of the affine transformation are determined by gradient-descent searching.The rigid transformation and the affine transformation use the same set of landmarks to formulate the similarity measures. Let Ba. The superior source landmarks are placed over the superior part of the contour of the Atlas head mask Ha. Their extraction is very similar to the extraction of those used in the MRI/CT registration, but they are much more sparsely distributed to reduce the computational overhead.The source landmarks used in the thin-plate spline transformation include several parts. The two primary parts are the inferior source landmarks and the superior source landmarks. Inferior source landmarks are distributed over the inferior part of the contour of the Atlas brain mask Hm and the line between the superior source landmark and the subject brain center. We have two ways to obtain the inferior target landmarks according to their locations. The first method is the same as that obtaining the superior target landmarks and works for the posterior part of the inferior landmarks that are distributed around the cerebellum. For the second method that works for the anterior part of the inferior landmarks, we compute a field of distance from the contour of the brain mask Bm, and each inferior target landmark, which should be on the contour of Bm, is obtained by tracing from the corresponding source landmark following the gradient of the distance field.The corresponding target landmarks in the subject head are determined automatically after the rigid and affine transformation are performed. For each superior source landmark, the corresponding superior target landmark is obtained as the intersection of the contour of the subject head mask Si and its mate landmark Si′, we have a distance diS; for each pair of the superior target landmark Ti and its mate landmark Ti′, we have a distance diT. A global ratio is then defined as diT/diS and its optimum is determined as the highest value so that the majority (such as 90%) of the superior target mate landmarks are off the mask Bm. For the few superior target mate landmarks that are within Bm according to the global ratio, the global value is decreased locally so that they are off the mask Bm.In the procedure of the registration, the Atlas skull's thickness is adjusted according to an estimated ratio of the subject skull's thickness to the Atlas skull's thickness. This is achieved by using another set of source landmarks and another set of target landmarks. For each superior source landmark, we determine its “mate landmark” over the inner side of the Atlas skull and on the line between the superior source landmark and the Atlas brain center. For each superior target landmark, there is also a “mate landmark” on the line between the superior target landmark and the subject brain center. For each pair of the superior source landmark For each posterior inferior source landmark, we determine its mate landmark over the outside of the Atlas skull and on the line between the source landmark and the Atlas brain center. For each posterior inferior target landmark, we also have a mate landmark on the line between the target landmark and the subject brain center while its location is determined according to the global thickness ratio described above. In order to further improve alignment performance, there are few other landmarks such as the centers of the eyeballs. We have a specific algorithm to automatically detect the subject's eyeballs, but the details are not described here for the limit of space. When all source landmarks and all target landmarks are determined, the transformation for any point in the space can be calculated as shown in .Hm. For the thin-plate spline transformation, the target landmarks are the GPS points while the source landmarks are those points on the contour of Hm so that each one is on the line connecting the brain center and the corresponding target landmark.The EEG sensor positions are automatically localized in 3D with the Geodesic Photogrammetry System (GPS). GPS captures images from 11 cameras in fixed positions to allow identification of the sensors in the images and then computation of the 3D coordinates . The AtlThe first automatic step of the MRI-to-GPS registration can be replaced by a transformation with the user providing three fiducial landmarks on the MRI space: the nasion and the left and right periauricular points (where the jaw hinge meets the skull). BrainK is able to stretch the MRI so that the user-specified landmarks match the corresponding ones in the GPS space. This initial state can be directly fed to the thin-plate spline transformation or it may pass through an intermediate affine transformation. BrainK also allows the user to specify only the nasion landmark and does rigid transformation in the first step while keeping the corresponding nasion landmarks matched.The Atlas-to-GPS registration is very similar to the MRI-to-GPS registration, except that the Atlas typically has greater geometric differences from the GPS than the subject's own MRI. The steps are similar to those for the MRI-to-GPS transformation just described. For the MRI-to-GPS registration, however, if they are to be used, the fiducial landmarks have to be manually marked on the MRI, whereas the fiducials are included in the digital Atlas dataset for the Atlas-to-GPS registration. In addition, one more fiducial landmark, the vertex, is used to accommodate the greater geometric difference between the Atlas and the subject.The GPS-to-head registration can be seen as the inverse of the registration of MRI-to-GPS, but no thin-plate spline transformation is needed. The initial transformation, either automatic rigid transformation or the stretching in terms of fiducial landmarks matching, can put the GPS in a good initial state. If needed, an affine transformation can be performed to further improve the alignment. The final adjustment then moves each GPS point onto the scalp, if it is not on it yet, with an end result that is similar to finding the corresponding landmarks in the MRI-to-GPS registration.When GPS is not available, a set of generic sensor positions may be used, such as those from the appropriate Geodesic Sensor Net (channel count and size) selected from the database. This registration has two steps. First, the generic spherical sensor cloud undergoes a translation transformation so that the center of the sensor point cloud overlaps with the center of the brain. Second, in a procedure that matches the physical conformation of the geodesic tension structure to the head, each sensor is mapped to a point on the scalp by the ray emanating from the brain center to the sensor's position.The well-known marching-cube isosurface algorithm  is perfo triples  are distributed evenly through the cortical gray matter.The dipole tessellation operates in two modes. If the individual's MRI is available, the cortical surface is extracted and tessellated into patches, and a dipole is fit to each patch. If it is not available, then dipole graph partition [ vertex corresponds to a triangle and each edge incident to two vertices corresponds to the adjacency between the two corresponding triangles. The graph partition algorithm then divides the entire graph into a set of parts, with the constraints that (1) the mesh is divided into the same number of patches as the desired number of oriented dipoles, (2) all patches are very compact , and (3) all patches have similar areas. The graph partition library Chaco [Whereas a trivial algorithm can distribute the triples, the even distribution of the oriented dipoles over the cortical surface is formulated as a sophisticatedartition  problem.ry Chaco  is used Given the mesh partition performed as above, an equivalent oriented dipole is then defined for each patch, generated at the center of the patch, with its orientation defined by the mean of all surface normals of the triangles in the patch, weighted by their areas. This equivalent dipole thus reflects the orientation of the electrical source if the entire patch was synchronously active, thereby serving as a useful approximation for the resolution of cortical activity that is set by the graph partitioning decision. The cortical surface and the oriented dipoles are displayed in The Talairach transformation component brings all the data, the segmentation, the cortical surfaces, the sensors, and the dipoles, into the standard Talairach space , 28. TheBrainK provides certain algorithms, such as skull bone density (X-ray CT) image registration and cortical surface dipole tessellation, which are specifically important to model the electrical properties of the human head. As described below in In addition to these somewhat unique capabilities, BrainK implements certain tissue segmentation and surface extraction algorithms that are useful in medical image processing generally and that can be evaluated in comparison with existing software that has become well known in the neuroimaging research community. Examples of the software examined here are FreeSurfer, SPM, FSL, and BrainVisa. To assure accuracy for medical applications, validation of BrainK's unique algorithms against well-known reference software not only is scientifically useful but also forms an integral part of medical quality verification and validation. In addition, medical use requires that the functionality is fast and intuitive, such that BrainK must meet standards of efficiency and ease of use.Our current evaluation work pays particular attention to brain tissue segmentation due to the fact that the accuracy of brain tissue segmentation is critical for a variety of medical and neurological applications such as source localization and cortical dysplasia detection.Below is a brief description of the four brain image analysis tools used for comparative performance evaluation. Although these tools provide varying functionality in neuroimage analysis, they all support automatic T1-weighted human brain MR image segmentation. Our comparative evaluation is focused on the image segmentation task.FreeSurfer , 30 is aSPM  is a statistical technique for testing hypotheses about functional imaging data . SPM alsFSL (the FMRIB Software Library) is a collection of functional and structural neuroimage analysis tools . For strBrainVisa , 37 is sThe evaluation was performed on three image datasets: a set of BrainWeb data with ground truth segmentation, a set of IBSR data with manually guided expert segmentation, and a set of real scans of subjects with either mild cognitive impairment or Alzheimer's disease.3. In the ground truth image, all voxels in the image are segmented into the following tissue types: Background, CSF, GM, WM, Fat, Muscle/Skin, Skin, Skull, Glial Matter, and Connective.The BrainWeb dataset is a group of 8 realistic T1-weighted MR simulated images with ground truth segmentation provided by BrainWeb, a simulated brain database , 43. AllA variety of noise levels and levels of intensity inhomogeneity ) are artificially introduced in the simulated images, as listed in 2 to 0.837 mm2. The MR images have been “positionally normalized” into the Talairach orientation, but all five tools performed on this group of data assumed that the images were not normalized. The MR images were also processed by the CMA bias field correction routines, but it is not guaranteed that the intensity inhomogeneity is completely corrected. All five tools therefore treated the datasets as if no bias field correction had been performed.The IBSR dataset is a group of 18 T1-weighted real MR brain datasets. Their manually guided expert segmentation is included in the Internet Brain Segmentation Repository (IBSR) supported by the Center for Morphometric Analysis (CMA) at Massachusetts General Hospital . The sliEach MR image was manually segmented into 44 individual structures including 3rd ventricle, 4th ventricle, Brainstem, and left and right: accumbens area, amygdala, amygdala anterior, caudate, cerebellum cortex, cerebellum exterior, cerebellum white matter, cerebral cortex, cerebral exterior, cerebral white matter, hippocampus, Inf Lat vent, lateral ventricle, pallidum, putamen, thalamus proper, ventral DC, and vessel.The 18 MR images represent various levels of image quality. To organize the evaluation by general image quality, we divided the images into two subgroups: the first 13 MR images with good quality and 5 more MR images with bad quality. Note that the ordering of the IBSR datasets is different from the original order. A map of the order we used to the original order is “1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 3, 4, 13.” For example, when we refer to the 3rd dataset, it is actually the 5th in the original order.3. The source of these datasets is the Neurobiology Research Unit [In addition to the BrainWeb and the IBSR datasets, which were used for both quantitative and qualitative evaluation, we also tested five tools on an auxiliary group of 8 real MR images scanned from subjects with minor cognitive impairment or those with Alzheimer's disease. The resolution of these datasets is 1.139 × 1.211 × 1.211 mmrch Unit  in the UIn this section, we present a quantitative evaluation on the segmentation accuracy, robustness, and computational efficiency of BrainK in comparison to other four packages. We use the widely used Dice metric , 46, 47 Let TP refer to the number of true positives, FP to false positives, and FN to false negatives. The Dice metric is then given byTP, FP, and FN are measured versus the manual segmentation of real MR datasets or the ground truth of simulated images. Note that when the segmentation is given as a probability between 0 and 1 for each image voxel for each tissue class (such as in the case of SPM), then TP, FP, and FN are calculated as the sum of the probabilities instead of discrete counting.For quantitative evaluation using Dice metric, we must decide the tissue type on which the metric is measured. BrainVisa only performs cerebrum segmentation while FSL and SPM segment the entire brain into CSF, GM, and WM without extraction of the cerebrum. FreeSurfer also performs segmentation on the whole brain, but it segments the brain into a greater number of tissue types . Currently, BrainK performs segmentation on the entire image, and it is able to differentiate cerebrum from cerebellum and one cerebral hemisphere from the other. In our quantitative evaluation, we must calibrate the segmentation of the five packages within a standard framework, so that common tissue types can be used for quantitative metric measurements.For the BrainWeb datasets, we calibrated the segmentation of five tools to the segmentation of cerebral WM and cerebral GM and measured the Dice metrics with respect to these two tissue types. To enable this, we manually partitioned the ground truth whole brain (WM plus GM) at the brainstem to extract the cerebral WM and the cerebral GM. Cerebral WM and cerebral GM also need to be extracted for the SPM and FSL segmentation results. We use a procedure (described in the next paragraph) that almost “perfectly” partitions the segmentation results based on the ground truth partition. For BrainVisa and BrainK, no transformation in the calibration is required. For FreeSurfer, we just need to relabel all cerebral cortex voxels and all subcortical voxels excluding cerebral WM as cerebral gray matter.v in FP-Brain, if it has a shorter path in FP-Brain to TP-Cerebrum than any paths in FP-Brain from v to TP-Cerebellum, then v is taken as a false positive (of WM or GM) in cerebrum; otherwise, it is taken as a false positive (of GM or WM) in cerebellum.Let TP-Cerebrum and TP-Cerebellum, respectively, denote the set of true positives of cerebrum and cerebellum in the segmentation of SPM and FSL. Let FP-Brain denote the entire false positives including those in cerebrum and cerebellum. The partition of the brain segmented by FSL and SPM is essentially the partition of FP-Brain into false positives in cerebrum and those in cerebellum, which is described as follows. For each voxel For the IBSR data, we calibrated the segmentation of five tools to the segmentation of cerebral cortex and cerebral WM and measured the Dice metrics with respect to these two tissue types. These quantitative metrics give an evaluation on the accuracy of the cortical surface reconstruction which depend on segmentation of cerebral cortex and cerebral WM and are irrelevant to segmentation of subcortical gray matter tissues. Since FreeSurfer explicitly labels cerebral cortex and cerebral white matter, we do not need to do any transformation in the calibration. The calibration of FSL and SPM first conducts the brain partition to extract the cerebral WM and cerebral GM. Given the set of cerebral WM and cerebral GM segmented by FSL, SPM, BrainVisa, or BrainK, we measured the Dice metrics with respect to cerebral cortex and cerebral WM in the way described below.We tested FreeSurfer on both the BrainWeb and the IBSR datasets in a fully automatic mode without any user intervention. An issue in collecting FreeSurfer segmentation results is the production of the cerebral cortex mask. There is a so-called “aseg” image and a “ribbon” image and both of these record voxels are labeled as cerebral cortex. The “ribbon” data is what FreeSurfer suggests to use, but it has more false negatives than the “aseg” data. In contrast, the “aseg” image is an intermediate result, and it has more false positives than the “ribbon” data. We applied a simple morphological closing operation on the union of the cortex ribbon and the subcortical structures so that certain true cerebral cortex voxels labeled in “aseg” but missed in “ribbon” are covered. This procedure apparently improved the performance of the cerebral cortex segmentation. The Dice metric for the “aseg” image was 0.7353, for the “ribbon” image was 0.8009, and for the “closed” image was 0.8212.In our first batch of experiments with FSL, we let FSL automatically extract the brain and performed brain tissue classification on both the BrainWeb and the IBSR datasets. However, FSL generated poor results on the brain extraction and brain tissue classification on 6 of the IBSR datasets (dataset 5 to dataset 10). In our second batch of experiments, therefore, we used different parameters in FSL, obtained better brain masks for these datasets, and then repeated the brain tissue classification. The brain masks generated in the second batch of experiments were still not good enough. We therefore used the brain masks generated by FreeSurfer for the brain tissue classification in FSL. This delivered the best FSL brain segmentation performance on the 6 difficult IBSR datasets. The three batches of experiments on FSL show that the brain extraction algorithm of FSL is not robust on the IBSR datasets, but the brain tissue segmentation itself performed well given good brain masks. The performance of FSL on the 6 IBSR datasets with respect to the three batches of experiments is shown in  a priori that there is less intensity inhomogeneity in the image. Since the IBSR datasets were processed with bias field correction, the use of “medium regularization” rather than the default “very light regularization” improved the performance of SPM on almost all IBSR datasets. For the cerebral cortex, the mean Dice metric over the 18 ISBR datasets was 0.7609 for very light regularization and 0.7656 for medium regularization. The mean Dice metric for white matter was 0.8572 for very light regularization and 0.8707 for medium regularization. We used the best regularization for each dataset in our comparative evaluation.In our first batch of experimental tests with SPM, we used the default parameters and let SPM automatically perform brain tissue segmentation on the BrainWeb and the IBSR datasets. In the second batch of experiments, we changed the parameter “bias regularization” from the default “very light regularization” to “medium regularization” and reran SPM on the IBSR datasets. SPM is supposed to be used with greater bias regularization when it is knownWe tested BrainVisa on both the BrainWeb and the IBSR datasets automatically, with the exception that we manually specified landmark points including the AC point, the PC point, an interhemispheric point, and a left hemisphere point. BrainVisa produced an empty brain mask in the 9th IBSR dataset and was unable to generate brain masks for the 13th and the 18th datasets. In both cases, we set the Dice metrics to be 0.BrainK was tested on the BrainWeb and the IBSR datasets fully automatically.We collected the Dice metrics with respect to cerebral cortex  and cereOn average, BrainK performed best on cerebral cortex segmentation on all 18 IBSR datasets, good and bad, and on the 13 good datasets exclusively. In particular, BrainK's cerebral cortex performance is consistently better than the four other packages on the 13 good datasets except for the 4th dataset, where BrainK's performance is almost identical to the best, and the 12th dataset, where BrainK's performance is close to the best. The cerebral cortex segmentation performance of the five packages on the five bad datasets is similar, except that BrainVisa generated empty brain mask for the 18th dataset.t-tests across these 18 datasets. BrainK was more accurate at segmenting cerebral cortex with the ISBR datasets than BrainVisa (p < 0.02), SPM (p < 0.006), FreeSurfer (p < 0.0005), and FSL (p < 0.0009). If we omitted the three datasets on which BrainVisa failed, the comparisons again showed BrainK to be more accurate than BrainVisa (p < 0.002), SPM (p < 0.009), FreeSurfer (p < 0.001), and FSL (p < 0.002).To provide an estimate of the statistical significance of the comparison of BrainK with the other packages, we conducted paired comparison On average, FSL performed best on cerebral WM segmentation on the 18 IBSR datasets and on the 13 good datasets. However, BrainK's performance was very close to FSL's performance in this comparison. The performance of the five packages on the five bad datasets is similar, except that BrainVisa generated empty brain mask for the 18th dataset, and FreeSurfer gave particularly poor performance for the 14th dataset.We calculated the standard deviations of the Dice metrics on the IBSR datasets. Together with the mean Dice metrics, these values indicate the segmentation robustness of the five packages on MR images scanned from different subjects. Greater mean Dice metric and lower standard deviation indicate greater robustness with respect to segmentation accuracy.Two groups of standard deviations were calculated: one on the total 18 IBSR datasets and one on the 13 good IBSR datasets. BrainK demonstrated the lowest standard deviation of the software packages with respect to cerebral cortex on both the total 18 IBSR datasets and the 13 good IBSR datasets, as shown in As described in Among the five packages, FreeSurfer demonstrated the lowest performance variation over different noise levels. SPM and BrainVisa are similar in showing the highest performance variations over different noise levels. BrainK and FSL have medium performance variations over different noise levels, compared to the other three packages. Although FreeSurfer performed consistently over different noise levels, it also gave results with the lowest accuracy on average.N = 9% and IIH = 40%. This is due to a poor brain mask.For each of the four noise levels, we also tested the packages on images with two different IIH levels. All five packages gave little variation over different IIH levels. The only exception is for BrainVisa with It is worth noting that, in real MR scans, the intensity inhomogeneity may occur in various and unknown patterns. Such noise could occur together with other difficulties that may be not present in the simulated BrainWeb datasets. Therefore, we remark that our experiments with the BrainWeb dataset should not be taken to give a thorough and sufficient evaluation on the five packages with respect to the IIH robustness.The execution times of the fives packages tested on the IBSR datasets and the BrainWeb datasets are listed in In this section, we give a qualitative evaluation of the five packages based on the results of applying the packages to the IBSR datasets, the BrainWeb datasets, and the 8 pathological datasets with mild cognitive impairment or Alzheimer's disease. We first summarize and compare the segmentation functionalities of the five packages followed by the discussion of their automaticity. Finally, we present various segmentation abnormalities that we observed in the experiments. Bias Field Correction. Freesurfer, SPM, FSL, and BrainVisa all integrate a bias field correction procedure, either prior to tissue classification or combined with the classification. BrainK, on the other hand, does not need an explicit bias field correction because the relative thresholding method is robust to bias field in arbitrary patterns. Brain Extraction. FSL, FreeSurfer, and BrainVisa provide separate tools for brain extraction  prior to brain tissue classification, whereas SPM combines brain extraction with tissue classification. BrainK, on the other hand, performs brain extraction after tissue classification. Note that the brain mask generated by BrainVisa is meant to contain only GM and WM while the brain mask generated by FSL and FreeSurfer is meant to contain CSF as well as GM and WM. Tissue Classification. FSL and SPM segment the brain volume into three tissue types: CSF, GM, and WM. BrainVisa extracts cerebral WM and cerebral GM. BrainK classifies the entire region of interest into CSF, GM, and WM, extracts the GM and WM, and separates cerebellum from cerebrum. BrainVisa and BrainK also provide cerebral hemisphere partition. FreeSurfer segments a whole brain into 37 individual structures including cerebral cortex, cerebral WM, a set of subcortical structures, brainstem, and cerebellar structures. Cortical Surface Reconstruction. BrainVisa, FreeSurfer, and BrainK support cortical surface reconstruction, but FSL and SPM do not. A core requirement for cortical surface reconstruction is to insure that the topology of the cortical surface is correct. Multichannel Segmentation. It is worth noting that FSL and SPM12 provide multichannel segmentation such that T1- and T2-weighted images can be combined together for increased accuracy. However, T2-weighted images are often not available and when they are available, T2-weighted images often decrease the contrast between grey and white matter and hence lead to bad performance in practice.The following is a summarization and comparison on the main segmentation features of the five packages:All five packages support highly automatic brain segmentation, with little or no user intervention. FreeSurfer allows the user to start the cortical surface reconstruction without any intervention. In case the segmentation is not satisfactory, FreeSurfer supports interactive tools to allow the user to modify the brain mask and to add control points to improve the intensity normalization of WM, a procedure that is extremely important for the performance of FreeSurfer's whole brain segmentation. FreeSurfer also supports interactive tools for editing the final results generated by the automatic processing.k-means segmentation or a priori probability maps for initial segmentation and to guide the k-means segmentation with manual intervention.FSL also allows the user to start the segmentation without any intervention. In FSL, brain extraction and tissue classification are performed, respectively, by the BET and FAST algorithms. The BET performance substantially influences that of FAST. If the user is not satisfied with the brain extraction, FSL allows the user to select different parameters and rerun BET. However, our experiments with BET on the IBSR datasets and the pathological datasets show that BET cannot guarantee good brain extraction even with user intervention. FAST has custom options for the user to select whether to use the In SPM, brain segmentation can also be automatically started with the default parameters and SPM often generates good results. An important custom parameter of SPM is the one that controls the extent of bias field regularization. When any parameter is changed, the segmentation procedure has to be started over from scratch.BrainVisa requires the user to prepare the data by first specifying several landmark points including the AC point, the PC point, an interhemispheric point, and a left hemisphere point. When the data is prepared by the user, BrainVisa automatically performs segmentation. BrainVisa supports interactive tools for the user to edit the segmentation results.BrainK performs segmentation fully automatically, but the user also has the opportunity to tune the performance by changing key parameters such as the relative thresholds. Compared to the user intervention mechanisms in the other four packages, user intervention in BrainK is in the form of global parameter selection and has the following advantages. First, it is straightforward and requires little or no expertise to understand the meaning of the parameters and the criterion for selecting optimal ones. Second, it is very easy to operate by sliding a value bar. Third, it is very efficient and the user can obtain the effect of parameter selection in real time for such parameters as relative thresholds. Fourth, the parameters have global effect for segmentation and the user does not need to repeat similar operations for different local regions.In examining the quality of the results, we first attempted to understand why BrainK consistently achieved better performance than the other packages with respect to cerebral cortex segmentation on the 13 good IBSR datasets . We founThe contrasting cortex segmentation of the five packages is shown for a representative IBSR dataset in A common problem in FSL, BrainVisa, and SPM is the poor brain extraction. Some examples are shown in  a priori probability maps used in FreeSurfer.We also found some other interesting abnormalities with FreeSurfer, as shown in  Modal Image Pipeline, EGI's commercial software that incorporates the BrainK algorithms and workflows. Validation tests of physical tissue boundaries are conducted in the routine workflow for each head model construction. A first step in the FDM electrical lead field generation [At this point, BrainK has built functional electrical head models for more than 200 persons. Critical evaluation of BrainK's performance for generating segmented volumes and cortical surfaces continues in the applications of theneration , 50, EGIThere have been many applications of BrainK in research and development projects that have tested its capabilities and limitations. For example, although the skull Atlases are organized by age and gender in a database to minimize the warping required to fit an individual's MRI, tests have been conducted with skull CTs and MRIs of divergent shape , with goElectrical source analysis with children requires age-appropriate electrical head models. Infants particularly require unique models because of their small head size and highly conductive skulls. BrainK has proven effective in adapting to the unique challenges of processing infant MRIs and CTs. The FDM electrical head models created by BrainK allow accurate estimation of the delivery of current, as well as its measurement, from the dense electrode array. With BrainK's anatomical results providing the geometry for the head physics code (Poisson equations for electromagnetic propagation in a finite difference model or FDM), lead fields are generated in the Modal Image Pipeline head physics code, describing the propagation from cortical sources to head surface electrodes  and from head surface electrodes to cortical sources . The results are visualized on BrainK's anatomy and registered electrode positions in the reciprocity engine developed by Erik Anderson of EGI . In thisBrainK is a software package that implements certain tissue segmentation and surface extraction algorithms that are useful in scientific and medical image processing. It also applies certain algorithms, such as skull image registration and cortical surface dipole tessellation, which are specifically important to model the electrical properties of the human head. Accurate electrical head models are proving important for improving the source analysis of human electrophysiological activity with dense array technologies and for optimizing transcranial current delivery to modulate brain activity.


The algorithms are tested using both simulations and bench experiments on a three degree-of-freedom robotic test bed. They are also compared with a position-plus-derivative controller as well as a receding horizon model predictive controller that employs an extended Kalman filter algorithm for predicting future heart motion.In robotic assisted beating heart surgery, the control architecture for heart motion tracking has stringent requirements in terms of bandwidth of the motion that needs to be tracked. In order to achieve sufficient tracking accuracy, feed-forward control algorithms, which rely on estimations of upcoming heart motion, have been proposed in the literature. However, performance of these feed-forward motion control algorithms under heart rhythm variations is an important concern. In their past work, the authors have demonstrated the effectiveness of a receding horizon model predictive control-based algorithm, which used generalized adaptive predictors, under constant and slowly varying heart rate conditions. This paper extends these studies to the case when the heart motion statistics change abruptly and significantly, such as during arrhythmias. A feasibility study is carried out to assess the motion tracking capabilities of the adaptive algorithms in the occurrence of arrhythmia during beating heart surgery. Specifically, the tracking performance of the algorithms is evaluated on prerecorded motion data, which is collected  The present work is focused on a previously unexplored aspect of robotic-assisted beating heart surgery; namely, evaluating performance of robotic active relative motion cancellation (ARMC) control algorithms for heart motion tracking under atrial fibrillation (AF) induced arrhythmia conditions. The occurrence of arrhythmia during cardiac procedures cannot be ruled out, hence robots must be able to handle this case to be clinically useful. This paper is the first to acknowledge this essential problem of tracking heart motion in the presence of heart rhythm irregularities during arrhythmia.Background for further details). However, the studies under slow heart rate variations are limited There is substantial amount of literature in heart motion tracking studies for robotic-assisted beating heart surgery in-vivo heart motion data is acquired from a bovine model, where the data includes artificially induced atrial fibrillation rhythm irregularities. Then, the ARMC control algorithms are tested on this data using simulations and bench-top experiments on a hardware test-bed. The algotihms are further compared with an extended Kalman filter (EKF) based control algorithm and a position-plus-derivative (PD) control algorithm. Although the control algorithms are formerly described in The purpose of this study is to evaluate the feasibility of the ARMC control algorithms introduced previously in on-pump, or passively restrain the beating heart with mechanical stabilizers, off-pump, in order to cancel the biological motion of the heart during CABG surgery. However, using on-pump CABG surgery might cause significant complications that might occur during or after surgery, which includes long-term cognitive loss Coronary artery bypass graft (CABG) surgery requires surgeons to operate on blood vessels that move with high bandwidth. This rapid motion of heart makes it difficult to track these arteries by hand effectively Robotic-assisted beating heart surgery has been proposed to enable high dexterity surgical manipulations to be performed on a beating heart without mechanical stabilization. In robotic-assisted beating heart surgery conventional surgical tools are replaced with robotic instruments, which are directly controlled by the surgeon through teleoperation. In this system, the surgeon views the surgical site through a camera mounted on a robotic arm, which follows the heart motion, giving a stabilized view. The robotic surgical instruments also track the heart motion, canceling the relative motion between the surgical site and the instruments. As a result, the surgeon operates on the heart as if it was motionless, while the robotic system actively compensates the relative motion of the heart Due to the high bandwidth of the heart motion, it is necessary to employ model predictive feed-forward control algorithms, which rely on estimation of the future motion of the point-of-interest (POI) to achieve sufficient tracking accuracy Several approaches have been proposed in the literature for modeling and predicting the motion of a POI on the heart, including algorithms using harmonics models in vivo constant and slowly varying heart rate motion data. Our previous report showed that the tracking of POI motion is no longer the bottleneck since the necessary amount of root-mean-square (RMS) tracking error on the order of 100–250 µm is achieved. Furthermore, if the heart behavior slowly changes, then adaptive predictors are able to adjust to these changes quickly enough and yield good tracking results.In our previous work two least-squares-based prediction algorithms, namely one-step and generalized, using an adaptive filter to generate future position estimates, were introduced in vivo arrhythmia data by simulations and on a three degree-of-freedom (3-DOF) robotic test bed. Additionally, tracking results of RHMPC with an EKF  by electrically stimulating right atrium has been used in the literature for easy and reliable modeling of AF Data were processed offline and only filtering performed on the data (using the proprietary software provided with the system) was very limited removal of the outliers, which occasionally occur as a result of ultrasound echoing effects. Description of the sonomicrometer crystal locations on heart with respect to the left anterior descending artery (LAD) are given in in vivo study protocol was approved by the Cleveland Clinics Institutional Animal Care and Use Committee, and all animals received humane care in compliance with the Guide for the Care and Use of Laboratory Animals  and institutional guidelines.The The most important aspects of robotic surgical instruments in robotic assisted CABG surgery are accurately measuring and predicting the heart motion as they are instrumental in tracking and canceling the relative motion between the heart surface and surgical tools attached to the robotic manipulators. The rapid motion of the heartbeat component possesses demanding requirements on the control architecture of the robotic system in terms of the bandwidth of the motion that needs to be tracked. Feed-forward control algorithms, which rely on estimations of upcoming heart motion, need to be utilized to achieve sufficient tracking accuracy Two adaptive filter-based predictors used in this study provide an estimate of the immediate future of the POI motion over a prediction horizon to the feed-forward control algorithm. The best estimate is defined to be the one that minimizes the square of the estimation error, where the estimation error is the difference between the prediction and the observed value at that time. Once a method is established to predict the next observations, a sequence of future observations can be estimated. The implementations of the predictors parameterize a linear system to predict POI motion and rely on recursive least-squares (RLS) adaptive filter algorithms. The one-step predictor assumes a linear system relation between the consecutive samples in the prediction horizon, whereas the generalized method performs the parametrization of the linear system independently for each point throughout the horizon.The estimated trajectory of the POI on the heart is used as a feed-forward control signal in a model predictive controller The details of the RHMPC algorithm can be found in In the authors' previous work It is necessary to evaluate algorithms on some type of hardware testbeds. The algorithms were tested on a PHANToM Premium 1.5A haptic device, which is a 3-DOF robotic system. The PHANToM provides insight into the effectiveness of the algorithms on a desired system. Its lightweight frame and drive system allow for sufficient motion and speed to attempt to track the heartbeat signal. With its high bandwidth and low inertia features, the PHANToM robot possesses characteristics similar to those desired in a surgical robot that would be capable of heart motion compensation. Such a surgical robot would need to have lightweight links, low inertia design, and low-friction actuation system for tracking the heart with sufficient motion and speed. Such a prototype is currently being developed In the experimental setup, control algorithms were executed in xPC Target and run in a real-time kernel with a sampling time of 0.5 ms on a Intel Xeon 2.33 GHz Core PC. The nonlinearities of the test-bed system  were canceled independently from the controller  As sonomicrometer was employed to acquire heart motion data used in this study, the sensor measurement delays were not considered, while determining the 25 ms prediction horizon. The sonomicrtometry sensor operates at high update rates  is an important variable that was continuously monitored throughout the study. Yet, since the HR values may vary in each and every given setting, the high variability of the baseline heart rate was not considered to be a drawback of the experimental approach in choosing a bovine model. Similar to humans, the HR may vary per baseline and intra-operative hemodynamic conditions, but no interference and/or vagal tone dominance was observed during the studies. However, if the native cardiac pacing is altered, then the vagal tone and correlated effects may become more significant in heart failure model.in vivo models of arbitrary arrhythmia types is not practical, if not impossible. A more broad in vivo study would overcome the limitations of the employed method of artificially inducing arrhythmias, and would cover more diverse situations in CABG surgeries and so will evaluate the performance and robustness of the proposed method in a wide range of conditions. Additionally, the applicability of the proposed system to specific arrhythmia patterns would be considered in further studies as a part of the development and safety control evaluation.We also note that the presented study is not intended to evaluate the tracking performance of the algorithms on each and every type of atrio-ventricular arrhythmia. Rather, by simulating one particular type of arrhythmia from a wide variety of patterns, we aim to show feasibility of employing adaptive estimation algorithms for heart motion tracking in the occurrence of arrhythmia during beating heart surgery. The underlying electro-physiological mechanisms of arrhythmia are very complex. Atrio-ventricular contraction is affected by multitude of factors, such as, changes in the function of ion channels in atrial cells. The complex pattern of electrical impulse propagation within atrio-ventricular tissues is also determined by the frequency of stimulation and summation of waveforms. As such, constructing One way to improve tracking quality is to incorporate other types of data into the estimation scheme. One such possibility is to include the electrocardiogram (ECG) signal into the observations. In this way, the predictor is able to use the electrical signals that activate heart contraction in order to improve the prediction as in in vivo validation of the proposed concept. Merging the sensor data from multiple position sources would increase accuracy of the motion estimation and improve tracking results. Adding more mechanical sensors that measure heart motion would improve the measurement precision and help to resolve calibration problems of the sonomicrometry system.Future works will also include multi sensor fusion where complementary and redundant sensors will be used for superior performance and safety, e.g., a vision based sensor system could be used as a secondary sensor for the


In other words, the output of the EMD array highlights contours of nearby objects. This conclusion is largely independent of the scale over which EMDs are spatially pooled and was corroborated by scrutinizing the motion energy profile after eliminating the depth structure from the natural image sequences. Hence, the well-established dependence of correlation-type EMDs on both velocity and textural properties of motion stimuli appears to be advantageous for representing behaviorally relevant information about the environment in a computationally parsimonious way.Knowing the depth structure of the environment is crucial for moving animals in many behavioral contexts, such as collision avoidance, targeting objects, or spatial navigation. An important source of depth information is motion parallax. This powerful cue is generated on the eyes during translatory self-motion with the retinal images of nearby objects moving faster than those of distant ones. To investigate how the visual motion pathway represents motion-based depth information we analyzed its responses to image sequences recorded in natural cluttered environments with a wide range of depth structures. The analysis was done on the basis of an experimentally validated model of the visual motion pathway of insects, with its core elements being correlation-type elementary motion detectors (EMDs). It is the key result of our analysis that the absolute EMD responses, i.e., the  Knowing the spatial structure of the surroundings is of crucial importance for many animals, especially if the environment is complex and cluttered. Spatial information is relevant for solving tasks such as collision avoidance, targeting objects, or landmark navigation. Depth information based on far-range mechanisms is of particular importance during fast locomotion in which animals often need to respond to objects when these are still beyond the range of close-up depth-sensing systems, such as stereoscopic vision or tactile sensing  and one with a constant depth for all directions . In the latter case, the images of the sequence are optically distorted the more the camera was displaced from the center of the sphere. Therefore, we only used the responses obtained in the center of the sphere for comparison because the corresponding images were virtually the same in depth-equalized and full-depth version. At this point of the simulated trajectory the only difference between the full-depth and depth-equalized image sequence is the depth distribution.Hence, we had two image sequences for each flight, one with the original depth structure , then a translational phase at 1 m/s (2) using the image sequence as input, then another 180° turn (3) and finally a translation backwards (4) which closes the movement loop. Rotations were rendered according to the dynamics of real blowfly saccades  and a high-pass filter (τH = 20 ms).with lp in each half-detector was modeled by a temporal first-order low-pass filter with a time constant of τlp = 40 ms  of the visual field:For comparison with environmental features (see below) the motion energy was determined for each retinotopic unit by taking the length of the motion vector given by the combination of the responses of a pair of the horizontal hEMD and the vertical vEMD at a given location : Figure Moreover, the depth estimation obtained in this way is affected by a systematic error. This error was determined by applying the described method to a movie generated by computer graphics where every distance to objects in the environment was known Figure . The errI  of all pixels  of the local region divided by the mean brightness I of the same region  of objects by taking the reciprocal of the corrected depth for each pixelTo investigate what information about the spatial layout of natural environments is represented by the visual motion pathway and, in particular, at its output level we simulated model responses to image sequences obtained on straight tracks in 37 different outdoor environments. We will show a sample image sequence at a given instant of time for the transformations of the visual input along the motion pathway to motivate what aspects of these transformations will be addressed quantitatively for all image sequences.The example image sequence obtained in a forest environment Figure  is transIt was already at the level of the temporally band-pass filtered activity distribution at the output of the peripheral visual system that mainly the edges of the nearby trees led to positive or negative responses depending on the polarity of brightness change at a given location in the visual field as a consequence of motion. In contrast, the distant parts of the scenery lead to only small responses Figure . This diIn case of the depth structure being equalized, all contrast edges irrespective of their nearness lead to visible responses at both the band-pass filter level and especially at the motion detector output. Without a differentiated depth structure all edges in a scenery move at the same angular velocity across the visual field. Thus the background contours move at a much higher velocity than during translation through the corresponding environment with a pronounced depth structure. In the chosen example, the strongest responses to the depth-equalized image sequence are found in the background as a consequence of high-contrast edges being present there  reflecting a large scatter between different natural sceneries. This scatter can partly be explained by the large noise in the depth maps, but mainly by the specific properties of the individual sceneries. For instance, image sequences without nearby objects, but a nearby ground consisting of sand with a very low contrast only lead to small EMD responses. In this case, the correlation is small as a consequence of the low signal-to-noise ratio of the nearness estimates.The hypothesis that the activity profile at the output of arrays of EMDs represents the contrast-weighted nearness or, in other words, the contrast borders of near objects was tested quantitatively for all motion sequences. Plotting the motion energy vs. the contrast-weighted nearness on a double-logarithmic scale reveals a roughly linear relationship between both parameters for the individual forest example mentioned above Figure  and, on R2 correlation value fluctuates over time around a kind of plateau level. During saccade-like rotations R2 completely drops toward zero. After the onset of translatory motion, it takes some time for the correlation value to reach its final intersaccadic level after the onset of translatory movement . However, since the nearness is constant for all directions, it does not affect the point-to-point correlation.The focus of expansion (FOE) and the focus of contraction (FOC) are singularities in the translatory optic flow field where depth cannot be extracted because at these locations there is no retinal image flow. Moreover, since the small retinal velocities close to these singularities might be more affected by noise than the larger velocities in other parts of the visual field, they are likely to reduce the correlation values between motion energy and contrast-weighted nearness. Therefore, we correlated both parameters after excluding the FOE, the FOC, and regions of variable size around these singularities. Moreover, in complementary tests, we kept only those regions around the singular points for correlation.Removing the FOE and the FOC as well as the surrounding areas does not increase the correlation at all Figure . On the What are the consequences of spatial pooling on the correlation of motion energy and contrast-weighted nearness? We expected some increase with spatial pooling because we observed, for instance at the edges of trees, small shifts between pixels with large contrast-weighted nearness values and the corresponding pixels of high activity in the motion energy profile Figure . TherefoSpatial pooling over a small range already increases the reliability of the local contrast-weighted nearness information to a great extent . Here, the animals essentially move straight for more than 80% of flight time and change their direction by interspersed saccadic turns of variable amplitude. Since translation velocity does not change much during intersaccadic intervals (Schilstra and van Hateren, The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.