The feasibility of automated individual crop plant care in vegetable crop fields has increased, resulting in improved efficiency and economic benefits. A systems-based approach is a key feature in the engineering design of mechanization that incorporates precision sensing techniques. The objective of this study was to design new sensing capabilities to measure crop plant spacing under different test conditions . For this study, three different types of optical sensors were used: an optical light-beam sensor (880 nm), a Light Detection and Ranging (LiDAR) sensor (905 nm), and an RGB camera. Field trials were conducted on newly transplanted tomato plants, using an encoder as a local reference system. Test results achieved a 98% accuracy in detection using light-beam sensors while a 96% accuracy on plant detections was achieved in the best of replications using LiDAR. These results can contribute to the decision-making regarding the use of these sensors by machinery manufacturers. This could lead to an advance in the physical or chemical weed control on row crops, allowing significant reductions or even elimination of hand-weeding tasks. Precision agriculture requires accurate plant or seed distribution across a field. This distribution is to be optimized according to the size and shape of the area in which nutrients and light are provided to plant to obtain the maximum possible yield. These factors are controlled by the spacing between crop rows and the spacing of plants/seeds in a row . For manThe current designs of vegetable crop transplanters and seeders utilize several uncoordinated planting modules mounted to a common transport frame. These systems use sub-optimal open-loop methods that neglect the dynamic and kinematic effects of the mobile transport frame and of plant motion relative to the frame and the soil. The current designs also neglect to employ complete mechanical control of the transplant during the entire planting process, producing an error in the final planting position, due to the increased uncertainty of plant location as a result of natural variations in plant size, plant mass, soil traction and soil compaction .Accurately locating the crop plant, in addition to allowing automatic control of weeds, allows individualized treatment of each plant . Seeking to ensure minimum physical interaction with plants , different remote sensing techniques have been used for the precise localization of plants in fields. For these localization methods, some authors have decided to address automatic weed control by localizing crop plants with centimetre accuracy during seed drilling  or transToday, one of the biggest challenges to agricultural row crop production in industrialized countries is non-chemical control of intra-row (within the crop row) weed plants. Systems such as those developed by Pérez-Ruiz et al.  or the cSpatial distribution and plant spacing are considered key parameters for characterizing a crop. The current trend is towards the use of optical sensors or image-based devices for measurements, despite the possible limitations of such systems under uncontrolled conditions such as those in agricultural fields. These image-based tools aim to determine and accurately correlate several quantitative aspects of crops to enable plant phenotypes to be estimated ,14.Dworak et al.  categori-To design and evaluate the performance of multi-sensor platforms attached to a tractor (a UC Davis platform mounted on the rear of the tractor and a University of Seville platform mounted on the front of the tractor).-To refine the data-processing algorithm to select the most reliable sensor for the detection and localization of each tomato plant.Based on the premise that accurate localization of the plant is key for precision chemical or physical removal of weeds, we propose in this paper a new methodology to precisely estimate tomato plant spacing. In this work, non-invasive methods using optical sensors such as LiDAR, infrared (IR) light-beam sensors and RGB-D cameras have been employed. For this purpose, a platform was developed on which different sensor configurations have been tested in two scenarios: North America  and Europe . The specific objectives, given this approach, were the following:To develop a new sensor platform to measure the space between plants in the same crop row accurately, laboratory and field tests were conducted in Andalucia (Spain) and in California (USA). This allowed researchers to obtain more data under different field conditions and to implement the system improvements required, considering the plant spacing objective. These tests are described below, characterizing the sensors used and the parameters measured.IR light-beam sensors  were used in two configurations: first as a light curtain , was used in the laboratory and field testing platforms to generate a high-density point cloud on which to perform the localization measurements. Its main characteristics are summarized in A Kinect V2 commercial sensor , originally designed for indoor video games, was mounted sideways on the research platform during field trials. This sensor captured RGB, NIR and depth images (based on time-of-flight) of tomato plants, although for further analysis, only RGB images were used for the validation of stick/tomato locations obtained from the LiDAR scans, as detailed in −1. For the odometry system, the shaft of an incremental optical encoder  was mounted so that it was attached directly to the gear shaft and used to measure the distance travelled, thus serving as a localization reference system. Each channel in this encoder generates 256 pulses per revolution, providing a 3-mm resolution in the direction of travel. The data generated by the light-beam sensors and the cumulative odometer pulse count were collected using a low-cost open-hardware Arduino Leonardo microcontroller  programmed in a simple integrated development environment (IDE). This device enabled recording of data that were stored in a text file for further computer analysis. Several repetitions of the tests were made on the platform to optimize the functions of both light-beam and LiDAR sensors. From the three possible LiDAR orientations, lateral scanning was selected for the field trials because it provided the best information on the structure of the plant, as concluded in [To maximize the accuracy of the distance measurements obtained by the sensors, an experimental platform was designed to avoid the seasonal limitations of testing outdoors. Instead of working in a laboratory with real plants, the team designed and created model plants see  using a luded in . In lab The initial tests, performed in Davis, CA (USA), were used to assess the setup of the light-beam sensor system and detected only the stem of the plants rather than locating it within a local reference system. Once the tomato plants were placed in the field, tests were conducted at the Western Center for Agriculture Equipment (WCAE) at the University of California, Davis campus farm to evaluate the performance of the sensor platform for measuring row crop spacing. For this test, an implement was designed to house the sensors as follows. The same IR light-beam sensor and encoder, both described in To continue the study of plant localization in a different scenario, additional experiments were designed at the University of Seville, in which a refinement of the LiDAR sensors and data processing were performed. These tests were conducted on several lines of tomato plants manually transplanted from trays, with the plants placed with an approximate, though intentionally non-uniform, spacing of 30 cm. Two of these lines were analysed further, one with 55 tomato plants and the other with 51, and a line of 19 wooden sticks was also placed to provide an initial calibration of the instruments. Due to the initial test conditions, where tomato plants were recently transplanted and had a height of less than 20 cm, the team built an inverted U-shaped platform attached to the front of a small tractor  The encoder proved to be sensitive to vibrations and sudden movements, so it was integrated into the axis of rotation of an additional wheel, welded to the structure and dampened as much as possible from the vibrations generated by the tractor. In addition, the team had to reduce the slippage of the encoder wheel on the ground to avoid losing pulses; (ii) Correct orientation of the sensors was also key because the mounting angles of the LiDAR sensors would condition the subsequent analysis of the data obtained in the scans and the determination of which data contributed more valuable information; (iii) The speed of the tractor should be as low as possible remain uniform (during the test the average speed was 0.36 m/s) and maintain a steady course without steering wheel movements to follow a straight path.To detect precisely and determine properly the distances between plants in both laboratory and field tests, the data provided by the sensors were merged and analysed.a)Pulse_distance_relation: This variable allowed us to convert the pulses generated by the encoder into the distances travelled by the platforms. In laboratory trials, the encoder was coupled to the shaft that provided motion to the 3D plants, and in the field, it was coupled to a wheel installed inside the structure of the platform. The conversion factors used for the tests were 1.18 and 0.98 mm per pulse for the laboratory and the field, respectively.b)Detection_filter: To eliminate possible erroneous detections, especially during field trials due to the interaction of leaves, branches and even weeds, the detections were first filtered. We filtered every detection that corresponded to an along-track distance of less than 4 mm while the sensor was active (continuous detection).c)Theoretical_plant_distance: The value for the theoretical distance between plants in a crop line. The value set during testing was 100 mm and 380 mm for the laboratory and the field, respectively.d)Expected_plant_distance: Expected distance between plants in a crop line was defined as the theoretical plant distance plus an error of 20%.Selection of Values for the Variables Used by the Programme for Detection:pulse_distance_relation) provided the distance in mm for each encoder value.Importing of raw data recorded by the sensors (encoder and existence “1” or absence “0” of detection by the IR sensors). The conversion factor (detection_filter). Thus, potential candidates were selected by registering the following: a)The distance at the start of the detection;b)The distance at the end of the detection;c)The distance travelled during detection and (iv) the mean distance during the detection, which was considered the location of the stem of the plant.Data were filtered by removing all detections whose length or distance travelled, while the sensors were active, was less than the set value (a)expected_plant_distance), we considered this candidate as a potential new plant, registering in a new matrix: the number of the plant, the detections that defined it, the midpoint location and the distance from the previous potential plant.If the evaluated distance was greater than the value set (b)expected_plant_distance), plant candidate data was added to the previous potential plant, recalculating all components for this potential plant. The new midpoint was considered the detection closest to the theoretical midpoint.If the evaluated distance was less than the set value  was similar to that described in . The algFor the analysis of the data obtained from the LiDAR, it is important to mention the high complexity of its data, in both volume and format, compared with those data obtained by the light-beam. This is reflected in the following section, which explains the proposed methodology for obtaining both the aerial point clouds of the tomato rows referenced to the encoder sensor and the tomato plant identification. This is a prerequisite for tomato plant localization. For this purpose, it was necessary to pre-process the data, followed by a transformation and translation from the LiDAR sensor to the scanned point.(i) Data pre-processing was performed at the LiDAR sensor. An off-line Matlab process was used with the actual field data collected during the field experiments. Data were filtered to eliminate false positives or those that did not contribute relevant information, considering only those detections with a distance greater than 0.05 m. Later, the resulting detections were transformed from polar to Cartesian coordinates using a horizontal orientation coordinate system as a reference.(ii) From the LiDAR sensor to the scanned point: transformations and data delimitation.a)The starting points were the Cartesian coordinates obtained using the horizontal orientation as a reference b)To integrate the scan from LiDAR into the platform coordinate system, a different transformation iDAR see . Each LiTo transform the horizontal LiDAR coordinates to the actual LiDAR orientation , the following steps were followed:z coordinate) were readjusted by subtracting the minimum obtained.Once transformed, the x translation was applied to coordinates obtained for the actual LiDAR orientation. The encoder values recorded at each scan time were used to update the point cloud x coordinate related to the tractor advance. Additionally, the height values  Aerial Point Cloud Extractiona)Gridding filter: Returns a downsampled point cloud using a box grid filter. GridStep specifies the size of a 3D box. Points within the same box are merged to a single point in the output see .b)pcfitplane : This MaThe aerial point data cloud was extracted using a succession of pre-filters. First, all points that did not provide new information were removed using a gridding filter, reducing the size of the point cloud. A fit plane function was then applied to distinguish the aerial points from the ground points. In detail, the applied pre-filters were:(ii) Plant Identification and Localization• Plant Clustering○An initial number of clusters: Floor  + 1) × 2○The squared Euclidean distance for the centroid cluster. Each centroid is the mean of points in the cluster.○The squared Euclidean distance measure and the k-means++ algorithm were used for cluster centre initialization.○The clustering was repeated five times using the initial cluster centroid positions from the previous iteration.○Method for choosing initial cluster centroid positions: Select k seeds by implementing the k-means++ algorithm for cluster centre initialization.A k-means clustering  was perfmin_distance_between_plants, the process was repeated by reducing the number of clusters by one  Plant LocationCentre of the clusterLocation of the lowest point at each plant cluster○By dividing the aerial plant data in slices defined by “histogram jumps”, the x limits on the maximal number of counts were obtained. For the z limits, data belonging to the bottom half of the aerial point cloud were considered.○The remaining data inside these limits were used to obtain a line of best fit, which was considered the stem line.○Plant location was defined as the intersection between the stem line and the corresponding ground line obtained previously from the MSAC function.Intersection of the estimated stem line and ground lineThree different plant locations were considered:x-axis, corresponding to the tractor advance). To obtain the distance between the stems of two consecutive plants using the Kinect camera, it is necessary to characterize the plants correctly and then locate the stems with RGB images from the Kinect camera. This characterization and location of the stem was conducted as follows: a sequence of images of the entire path was obtained (~250 images in each repetition), where the camera’s shooting frequency was established steadily in at 1-s intervals. Obtaining the string with the timestamp of each image was a key aspect of the routine developed in Matlab, as this string would later be used for integration with the LiDAR measurement. The relationship between the timestamp and its corresponding encoder value was used to spatially locate each image ((i)According to , the fir(ii)Next, the image segmentation step of an object-based image analysis (OBIA) was performed to generate boundaries around pixel groups based on their colour. In this analysis, only the green channel was evaluated to retain most of the pixels that define the plant, generating a mask that isolates them c and cla(iii)Each pair of consecutive images was processed by a routine developed in Matlab using the Computer Vision System Toolbox. This routine performs feature detection and extraction based on the Speeded-Up Robust Features “SURF” algorithm  to generImage processing techniques applied in the characterization of tomato plants from Kinect images generally followed these steps: (1)Calculating the distance in pixels between two consecutive images, as well as the encoder distance between these two images, the “forward speed” in mm/pixel was obtained for each image;(2)For each image, and considering that the value of the encoder corresponds to the centre of the image, the distance in pixels from the stem to the centre of the image was calculated. Considering whether it was to the left or to the right of the centre, this distance was designated positive or negative, respectively;(3)The stem location was calculated for each image using the relation shown in Equation (2) below;(4)Plant location was obtained for each image in which the plant appeared; the average value of these locations was used to calculate the distance between plants:Once the plant was identified in each image, the location of the stem of each plant was defined according to the following subroutine: Several laboratory tests were conducted to properly adjust and test the sensors. As an example, Three laboratory tests corresponding to three different stages of the light-beam sensor adjustment process are shown in The histogram in this figure represents the estimated distances between plant stems, where the average distance is 102.5 mm when the real distance between plants was 100.0 mm. The average standard deviation for the three trials was 2.2 mm. For test 3 . The system was intended to detect the presence of plant stems correctly and to distinguish between the foliage and the stem. In addition, this system was used to measure the distances between plants accurately, providing valuable information for future herbicide treatments. Because the use of the laser scanner for detection implies the generation of a very dense cloud of points, in which not all data are relevant, an initial filtering was performed as explained above. Regarding plant detection, x-axis on the plot) provided by the encoder is given in mm. Each plant/stick detected is marked with different colour . Related to the data presented in As explained in Plant locations obtained from the LiDAR and Kinect data are shown in The centre of the cluster is marked in green, the lowest point in blue, and the intersection in red. Encoder stick/plant locations by the Kinect image (location ± Std.) are both marked in black lines, and their distances are represented by black doted.The results obtained from the precise locations of the plants allow us to consider the viability of further methods capable of performing individualized treatments for each of the plants, or accurate agrochemical treatments to remove existing weeds between crop plants in the same row.A light-beam detection system was improved by reduction of the number of sensor pairs used. The reduction from three pairs to one pair had no effect on the desired plant detection results. In the field tests, 98% precision of detection was obtained, similar to that obtained in , showingBased on the methodology presented in the analysis of the data from the LiDAR and the Kinect sensors, different considerations can be established regarding the location of the same plant. According to the structure and morphology of the plant, it is assumed that the aerial part will not always be vertically in line with the stem . For this reason, one of three proposed locations can be established as the location of the plant: the aerial part, the cluster centroid of the filtered point cloud or the insertion of the stem. Depending on the type of treatment to be made, one of these locations could be more interesting to evaluate than the others. For example, if a variable foliar herbicide treatment is to be applied (discriminating between weeds and crop plants), the distance between the aerial parts of the plants should be given greater weight in the application system (to avoid applications on the crop plants and maximize efficiency by ensuring the herbicide is applied to weeds). In the case of physical weed-removal systems, as proposed in , prioritThe high volume of data generated by the more accurate sensors, such as the LiDAR used in this work, can be a hurdle for automatic weed detection machines when working in real time. However, it is important to emphasize the exponential growth of the processing algorithms available for the researcher, which can significantly reduce the time required for point cloud data analysis. Reducing the density of information necessary while continuing to give accurate information is an interesting subject for future work.The precise locations of the plants were determined using an encoder. The use of this type of sensor is vital to implementation of low-cost referencing. Correct localization and integration to enable determination of reliable reference of the movement is of great importance.Agricultural use of affordable electronic components will lower costs in the near future, but the authors conclude that these systems must be robust and provide a rapid response and processing times.A combination of optical sensors mounted on a frame on a tractor, which are capable of detecting and locating plants and utilizing ground-wheel odometry to determine a local reference system, was successfully developed and tested. Combining the use of affordable sensors (light-beam and Kinect) with more expensive ones (LiDAR), a laboratory setup platform and two field test platforms have been created. The following conclusions were drawn based upon the results of this research:


In the most recent report published by the World Health Organization concerning people with visual disabilities it is highlighted that by the year 2020, worldwide, the number of completely blind people will reach 75 million, while the number of visually impaired (VI) people will rise to 250 million. Within this context, the development of dedicated electronic travel aid (ETA) systems, able to increase the safe displacement of VI people in indoor/outdoor spaces, while providing additional cognition of the environment becomes of outmost importance. This paper introduces a novel wearable assistive device designed to facilitate the autonomous navigation of blind and VI people in highly dynamic urban scenes. The system exploits two independent sources of information: ultrasonic sensors and the video camera embedded in a regular smartphone. The underlying methodology exploits computer vision and machine learning techniques and makes it possible to identify accurately both static and highly dynamic objects existent in a scene, regardless on their location, size or shape. In addition, the proposed system is able to acquire information about the environment, semantically interpret it and alert users about possible dangerous situations through acoustic feedback. To determine the performance of the proposed methodology we have performed an extensive objective and subjective experimental evaluation with the help of 21 VI subjects from two blind associations. The users pointed out that our prototype is highly helpful in increasing the mobility, while being friendly and easy to learn. The report published by the World Health Organization (WHO)  in AugusIn order to acquire additional information about the near surrounding, VI people rely on traditional assistive elements such as the white cane or guide dogs . HoweverIn this paper, we propose a novel navigation assistant that facilitates the safe movement of VI people in urban scenes by using both sensors and computer vision technologies. The system is able to acquire information about the environment, interpret it and alert users about possible dangerous situations/obstacles. At the hardware level, the proposed framework is composed of a regular smartphone device, Bluetooth bone conduction headphones, ultrasonic sensors, an Arduino microcontroller and a flexible waist belt. Our solution can be characterized as low-cost and easy to wear because it is based on general public components available on the market, and does not require potentially costly, dedicated components. The major contribution of the paper concerns the fusion of two independent information sources, including ultrasonic sensors and the embedded video camera, which makes it possible to obtain a wearable assistive device that can reliably function in both outdoor and indoor scenarios. The framework is designed to detect and recognize, in real-time, both static and dynamic obstacles using a smartphone device as a processing unit. At the software level our major contribution concerns the effective integration, optimization and adaptation of different machine vision/learning techniques, while fully taking into account the information provided by ultrasonic module. Our system is carefully conceived to transmit warning messages fast enough for a VI user who is walking normally.The rest of the paper is organized as follows: In the last years various Electronic Travel Aid (ETA) technologies have been introduced. They were developed as a digital enhancement to the traditional white cane and aim at increasing the mobility of VI users. The following sub-sections describe and analyze the state of the art in the field, with systems based either on sensor networks or computer vision techniques.The sensor-based ETA systems are designed to collect environmental information and to transmit it to the VI user through a set of acoustic or haptic signals. One of the first ETA introduced in the state of the art was based on Mowat sensors , which eIn , two sonThe binaural sonar introduced in  detects The GuideCane approach that also uses an ultrasonic sensor to detect obstacles and an embedded computer to determine the optimal walking path was proposed in . The navThe EPFL system proposed in  uses a mIn , the CyAThe SUGAR system, recently introduced in , is an iThe analysis of the state of the art shows that the assistive devices based on sensorial substitution of human vision can be very effective when used in indoor environments or when employed to detect large, flat structures. However, in the case of outdoor urban scenarios the systems’ sensitivity becomes prohibitively high. Moreover, ETAs based on sonar are sometimes unable to identify with high confidence dynamic objects situated in the near vicinity of the user or other important features such as the borders of a sidewalk and stairs. Regarding the hardware architecture, the frameworks recently proposed in ,10 can bRecent advances in computer vision techniques have led to the rapid proliferation of assistive devices based on artificial intelligence, dedicated to outdoor/indoor navigation. In , the SmaIn our previous work presented in , a complA six degrees of freedom (6DOF) simultaneous localization and mapping (SLAM) solution embedded on a wearable device and using two stereo cameras is proposed in . The sysA navigation assistant using a head-mounted stereo-vision system is proposed in . By combIn , the proWith the developments of the Red, Green, Blue and Depth (RGB-D) cameras, different guidance systems ,18,19, hAn assistive device designed to recognize 3D objects is proposed in . At the The analysis of the state of the art shows that, until now, no method can fulfill to a satisfactory degree all the features required to facilitate the safe movement of VI people in an unknown outdoor environment. When comparing the sensor-based systems with frameworks relying on computer vision it can be observed that each approach presents advantages and limitations, but no single method is robust enough to replace the white cane. Starting from this observation, in this paper we propose a novel system that uses two sources of information from both ultrasonic sensors and a regular 2D video camera in order to acquire sufficient data for a comprehensive understanding of the scene. The proposed approach is described in the following section.The hardware components considered and integrated in the proposed device  include:In order to facilitate the safe movement of the VI people in an outdoor/indoor environment, at the software level, we introduce two novel modules able to detect and recognize possible dangerous static or dynamic obstacles existent on the walking path . The obsLet us now present the proposed obstacle detection module, which jointly exploits the video signal acquired by the smartphone camera and the information provided by the ultrasonic sensor, within the framework of an interest-point object detection approach, exploiting apparent motion analysis and integrating distance information.Let us first detail how the video signal is exploited for both static and dynamic obstacle detection purposes.The method starts by analyzing the first frame captured by the video camera of the smartphone device. We have evaluated several different point of interest extraction algorithms, including SIFT , SURF 22, BRIEF k candidate points from the entire image, without considering the spatial information  algorithm  has beenH between two successive video frames. For each relevant interest point 2L norm. In the ideal case, the estimation error  that satisfy the transformation. The remaining points are considered as outliers and correspond to foreground objects.The scene global motion is estimated using the RANSAC algorithm  that detThe user displacement affects significantly the video acquisition process, even in the case where the smartphone is attached on a waist belt. The video stream is unstable and presents cycling pan, tilt oscillation or panoramic distortions.H) is determined by using all the points of interest included in the current rectangle and only 25% of the points of interest from other regions, randomly selected. Next, each homographic matrix is used to determine its set of inliers/outliers. Finally, the interest points are labeled as belonging to an object motion if the estimation error is superior to a threshold In order to increase the robustness of the global motion estimation model we have proposed to analyze distinctively various areas of the frame. Multiple homographies have been computed over the same image by using different sets of points of interest. More precisely, the image is divided into four equal, non-overlapping rectangles . Then, foutlier points are further used to recognize different types of objects existent in the scene.Because of the camera motion induced by the user’s movement, even static foreground objects present an apparent motion that can help distinguish them from the background. In addition, various dynamic obstacles, including vehicles, bicycles or pedestrians can be identified by performing the same motion analysis process, detailed in the following paragraphs.2L distance between different samples in order to form groups. Based on this observation, the proposed algorithm integrates the following non-linear transform that changes the motion vectors parameters from polar coordinates back to the 2D Cartesian space:d is the radial coordinate that incorporates the magnitude information. The value of A clustering algorithm that uses as input the points of interest with the associated motion vectors is first applied. However, it is not feasible to use directly the motion vectors’ magnitude and orientation expressed in polar coordinates because the angular coordinate has a circular range of values, varying from 0 to 2k-means algorithm is applied. The algorithm aims at partitioning the outlier points of interest into a set of groups k is the maximum number of clusters that we have retained from the observation data. In our experiments we fixed k to 10 classes.Next, in order to identify the various moving objects, a p, an associated neighborhood region The quality of the clusters is verified by analyzing the spatial distribution of the points of interest in each class. For each cluster, its associated binary mask c is consions cf. . The iniIn a general manner, the proposed visual object detection framework proves to be highly effective in identifying fast moving objects. However, the module becomes sensitive in some particular conditions, most notably when: (a) the video camera experiences sudden movements; (b) the video stream is blurred and (c) the object moves on a parallel path with the VI user. In such cases, the motion vectors cannot be estimated efficiently, which penalizes the related detection performances.On the other hand, large obstructions  cannot be detected because the object represents more than a half of the video frame size. In such cases, the system cannot distinguish correctly the background from the foreground. In order to overcome such limitations we have considered a complementary obstacle detection module based on ultrasonic sensors. In addition, the sensor information makes it possible to reliably determine the distances to the detected obstacles.The proposed module is designed to detect and localize the obstacles situated on the walking path of the VI user by using an ultrasonic network architecture properly placed on the waist mounted belt. In order to optimize the sensor distribution on the belt, according to their capabilities of detecting obstructions, we have first evaluated their related performances, with the help of the following three different calibration objects: a cylinder (10 cm diameter and 85 cm the height), a rectangular box (40 cm width and 125 cm height) and a panel (180 cm width and 145 cm height). Such objects are commonly used in the state of the art [In order to evaluate the detection range of the ultrasonic sensor, all objects have been moved as follows: (1) from an outside position to the center of the sensors and (2) on the contrary, from the center of the sensors to the opposite outside perimeter. In both cases, an average speed of 1 m/s has been considered. The sensors performance was measured by gradually modifying the distance between the transducer and the considered object from 50 cm to 500 cm in 50 cm steps. For each considered distance, the detectable angles have been estimated. The obtained results are illustrated in The lowest performances have been obtained for the cylinder (±30 degrees and 350 cm), while the best results are achieved for the panel (±35 degrees and 450 cm). This behavior can be explained by the geometrical structure of the considered objects. From the experimental results presented in Such results need to be confronted with the requirements from real-life situations. We have observed that a majority of dangerous obstacles appearing in practice are situated at a distance less than 2 m and at an angle relative to the VI user walking direction ranging between ±50 degrees. This means that the sensor network needs to be designed such that it can ensure coverage of an angle up to 100 degrees.Based on these considerations, we have decided to use four ultrasonic sensors properly placed on the waist belt  in orderBy differentiating between the emitted wave and the measured echo, the microcontroller can estimate the distance between the VI user and the closest obstacle. The sensors are synchronized in order not to interfere. The information is transmitted as a Pulse-Width Modulation (PWM) signal to the receiver. The error estimation is less than 2 cm, which fits perfectly the application constraints. The microcontroller measures the width of the echo pulse and estimates the distances. This information is transferred with the help of the Bluetooth module to the smartphone device. Depending on the sensor that actually performs the detection, the object location within the scene can be estimated as follows: left, right, center-left or center-right of the person.The smartphone continuously receives information from the sensors through the Bluetooth module and transfers it to the computer vision module. This makes it possible to assign a distance to each motion cluster identified using the foreground detection method cf. . In ordeurgent if is situated in the near vicinity of the user (less than 2 m) otherwise it is marked as a regular obstruction. This rough classification helps to better prioritize the acoustic feedback messages depending on their relevance . Such an approach does not penalize the overall system performance because the warning messages are transmitted only for the closest, However, if the sensors indicate the presence of an obstruction in a certain area, this information is automatically transmitted to the acoustic feedback module even if nothing is detected by the computer vision algorithms. On the other hand, fast moving objects with a displacement speed of more than 7 m/s can affect the sensor performance. Therefore, in this case only the vision algorithms can predict the presence of a highly dynamic object in the scene or the existence of an obstruction out of the sensor covering range. For all these obstacles a default distance value of 5 m is assigned to. This value was selected based on the experimental results conducted. The maximum detectable distance of the MaxSonar LV EZ-0 sensor is less than 450 cm, whatever the test object involved. Thus, if an obstacle is identified only by the computer vision module it will be consider as situated outside the sensor action range. In order to keep coherence between the two detection modules, a 5 m distance is assigned for all objects situated outside the ultrasonic reliable detection range.In order to prioritize the acoustic feedback messages it is important to distinguish between the various obstructions detected and to semantically interpret them. In the following section we introduce a fast method that is able to classify in real-time the detected objects into four major categories corresponding to the most important types of obstructions that a VI user can encounter during navigation: vehicles, bicycles, pedestrians and static obstacles.The obstacle recognition framework involves an offline learning process that exploits a training image database structured into four major and generic categories of obstacles: vehicles, bicycles, pedestrians and static obstructions. The categories have been selected in order to correspond to most common types of dynamic/static objects encountered during indoor/outdoor navigation. The class of static obstacles is characterized by a high variability of instances such as: pylons, traffic signs, blocking rod; trees, garbage cans, bumps, walls, fences or bushes. Because the main constraint of our framework is to develop a real-time application we have decided to simplify the classification process to a relatively reduced set of categories (four) rather than perform a generic recognition task . The learning dataset is composed of 10,000 image patches selected from the PASCAL repository . Each can contiguous pixels brighter than the point being analyzed.The image patches extracted by the detection modules cf.  are mappThe extracted points are further described using the BRIEF algorithm that uses binary strings to characterize the informational content of a point of interest neighborhood. The descriptor is highly discriminative, uses a relatively small amount of bits and can be computed based on intensity tests. The differences between two descriptors can be determined by evaluating the Hamming distance (requiring a maximum storage capability of 256 bits), which is much faster than the traditional Euclidian distance.The interest point BRIEF descriptors need to be aggregated into a global image patch representation. One popular approach relay on the Bag of Words (BoW)  algorithThe VLAD image description can be regarded as a non-probabilistic variant of Fisher vectors that encode the distribution of interest points according to cluster centers. Compared with BoW VLAD can achieve very competitive retrieval performances with significantly smaller complexity and memory demands .k-means clustering algorithm. Here, C denotes the set of centroids, while X is the descriptor space. So, for each image I, N points of interest :Similar to BoW, but at a reduced scale, a codebook 2L normalized in order to have an equal contribution of each cluster in the final vector:To reduce the influence of bursty elements within images the power low normalization has been applied: d equal with the size of the local low level descriptor. Therefore, the size of V can be determined as d × K. In the experiments, we have employed a d value equal with 256 dimensions for the BRIEF descriptor and a vocabulary size K of 256 visual words.The global image representation using VLAD is given by the concatenation of all the residual vectors The obtained VLAD descriptor is further normalized to the unit length: echnique  that allThe final step of the image recognition module is given by the SVM training (offline process) and SVM prediction (online process). For the offline process, we have used a dataset composed of 10,000 image patches and their associated global VLAD descriptors. We have adopted a SVM classifier with bilities . For eacIn the online phase, for each image patch extracted by the obstacle detection module, a global representation using binary VLAD descriptors is constructed. This representation is applied as input to the trained SVM models. The one-versus-all classification method has been used in order to predict the image patch category. A patch is tested against all trained SVM and the one with the greatest margin and To satisfy the real-time constraint imposed to the application, the entire process has been parallelized on multiple threads. The number of threads is adaptively adjusted depending on the total number of objects detected in the scene. In addition, in order to deal with the situation when an image patch cannot be confidently assigned to any of the considered classes  will be transmitted before a warning regarding a pedestrian situated in the urgent area. After the discussions with the association for the blind involved in the project it was decided to give higher importance to approaching vehicles based on their speed of displacement and degree of danger. A possible collision with a fast approaching vehicle will have a higher impact over the VI user when compared to a possible collision with a walking pedestrian or an obstruction.In addition, in order to infer information about the object location within the considered environment, the set of acoustic signals is stereo-encoded. Thus, if the detected object is situated on the left side of the VI user, the warning message will be transmitted on the left helmet of the bone conducting headphones and vice-versa for obstructions located on the right side of VI person. For obstacles identified in front of the VI, the alarms are transmitted in both helmets. For far away objects situated at a distance more than 3 m, a higher confidence is given to the vision module, while for closer objects localized at a distances less than 3 m a higher confidence is given to the sensor module.In order to validate the proposed approach, an extensive objective evaluation of the system, presented in the following section, has been performed.The proposed assistive device has been tested mostly in outdoor environments because of the high complexity and diversity of static/dynamic objects. Regarding the computer vision-based obstacle detection module the video camera embedded on the smartphone has been used as an acquisition device. The video stream is acquired at a frame rate higher than 25 images with a resolution of 320 × 240 pixels. A testing database has been developed, including 20 items with the average duration of 10 min. When a ground truth data set is available, the obstacle detection/classification errors can be globally described with the help of two error parameters, denoted by P), recall (R) and F-norm (F) parameters, extensively used in the state of the art for evaluation purposes [Based on these entities, the object detection/classification performances can be globally evaluated using the precision . The recall measure can be interpreted as a measure of completeness, being inversely related to the second type errors . An ideal system should achieve both precision and recall scores close to 1.0. The recall and precision values can be combined into a single measure denoted by F (or F-score), which helps to determine the overall module performance, by considering as equally important the precision and recall scores.The precision For better visualization, the information gathered in The computer vision techniques return higher detection scores when compared with the ultrasonic approach. This behavior can be explained on one hand by the sensor sensitivity to highly dynamic objects  or hollow objects. The field of view of a regular video camera is superior when compared to the sonar range of action. As a consequence, the vision algorithms are able to detect the faster moving obstacles situated at longer distances (ranging from ten to twenty meters). For sonars, in the ideal case, the detection distance is limited to five meters.inliers and will be considered as part of the background/camera motion, while the outlier interest points  will contribute to the subsequent clustering process and will be interpreted as various obstacles existent in the scene.As expected, the vision methods are highly sensitive to the quality of the estimated motion vectors. Because the static obstacles are identified based on the camera apparent motion analysis, if the user stops suddenly, the system losses the motion information and cannot identify static obstructions. For large objects, that cover more than a half of the image size, the system returns a high number of missed detections. On the other hand, the sensor module proves to be highly robust to all the above situations. F-score, regardless of the obstruction type or nature .When the two detection modules are put to work in common, an important improvement is observed, with more than 6% on the Based on the analysis of the experimental results obtained, the following conclusions can be highlighted:In the second part, we have focused on the evaluation of the obstacle classification module that allows the semantic interpretation of the detected obstacles. The recognition system receives as input the image patches extracted using the vision-based object detection module. As mentioned in Concerning the acoustic feedback, for all image patches classified as Outliers only a BEEP warning is sent to the VI user. For objects identified only by the sensor module it is impossible to assign semantic information. In this case, in order to inform user about the presence of a possible dangerous obstruction a BEEP warning is sent into the bone conduction headphones.F-score measures are presented in The experimental evaluation of the recognition module was conducted on a set of 1190 image patches. The classification-related precision, recall and F-score superior to 82% for each of the considered categories. The lowest rates are obtained for the bicycle class. This behavior can be explained by the images used in the training phase. As indicated in The experimental results obtained validate the proposed approach, with a classification Regarding the computational complexity, the average processing time of the proposed framework when activating all modules  is around 100 ms, which leads to an average processing speed of 10 frames per second. The experimental results were performed on a regular Android smartphone device (Samsung S7). In terms of the battery consumption, the smartphone and the sensor module are simultaneously powered from an external battery of 16,000 mAh that allows running our system continuously for 5 h.In order to determine the performance of the proposed framework we have used for evaluation a set of 21 visually impaired people with ages ranging between 27 and 67 years. The main goal of testing was to gather information related to the following aspects: are the users able to start the application by their own? Can they safely navigate in a novel environment? Is it possible to avoid obstacles using the set of acoustic warnings? Is the system globally useful and can it complement the white cane?At the beginning of the testing phase the older VI people prefer to rely on other senses in order to complete the navigation scenario rather than listening to system acoustic signals. We have observed retention and mistrust to innovations. However, in the case of young VI users the system had a great impact and they expressed the willingness to use our framework frequently in daily activities.After an initial training phase, performed with the help of technicians or carriers, most participants expressed that the system is useful and easy to learn. Even the people without sufficient abilities to handle a smartphone device expressed strong interest in the proposed architecture.VI persons found the system wearable, friendly, light weighted and non-intrusive satisfying the hands-free and ears-free requirements. Even the older VI people consider the proposed system light weighted because the weight of all components summed together is inferior to 750 grams -152 grams, Bluetooth bone conducting headphones (AfterShokz Bluez 2)-41 grams, the microcontroller (Arduino Micro)-6.5 grams, the bluetooth device (Master Slave HC-05)-10 grams, 4 ultrasonic sensors (MaxSonar LV EZ-0)-4.23 grams, an external battery -308 grams and a Smartphone waist belt-200 grams).The various functions included in the architecture were well integrated and the warning messages are transmitted fast enough so users can walk normally.The usage of bone conducting headphones is appropriate because they allow hearing other sounds from the environment. The speech feedback is considered better than any vibro-tactile stimulation or an acoustic strategy based on beep patterns. The beeps can warn user about the proximity of an obstacle and not about the relative position or degree of danger. Moreover, inferring additional cognition over the scene is impossible.Most participants pointed out that the system is very useful when combined with the white cane, but they do not feel confident enough to use it as a standalone device. The white cane is designed to detect possible obstructions situated in the near vicinity of the user, but it requires an actual physical contact with the object. However, it is not able to provide additional information elements such as: the speed and type of object the user is encountering, the static or dynamic nature of the obstacle, the distance and time to collision. Even so, the white cane is the cheapest, the simplest and the most reliable element used as navigation aid. Because, the VI people are very familiar with the white cane they feel reticence and uncertain to completely switch to this assistive device, so any novel electronic travel aid (ETA) should be designed not to replace the cane, but to complement it in an intelligent manner, with additional functionalities. This is also the purpose of the proposed system.Users were satisfied with the acoustic feedback even when tested in crowded outdoor scenes with multiple moving objects and obstructions. Moreover, VI people were enthusiastic when they realized that the system is able to detect obstacles situated at head level.The testing was performed in various urban scenes for which VI people have no initial knowledge about the objects location, position or size. The proposed system was evaluated in various arbitrary locations and not under a specific environment. Each VI person was asked to walk on a test path and to use both the proposed device and the white cane. After a participant completed the task, an observer conducted a post-test debrief interview with users about the behaviors they observed during the testing. After discussions with the participants, the following conclusions can be highlighted:This paper has introduced a novel electronic travel aid that facilitates the safe navigation of VI people in urban scenes by using both sensors and computer vision technologies. At the hardware level, the proposed system is composed of a regular smartphone device, ultrasonic sensors, Bluetooth module, microcontroller, bone conduction headphones and a waist belt. At the software level, the major innovation of the proposed assistive device concerns the fusion of two completely independent sources of information: ultrasonic sensors and video streams. The system can identify with high confidence static or highly dynamic objects existent in the scene regardless on the object location, size or shape. In addition, in contrast with the most of the state of the art systems, the proposed prototype is able to semantically interpret the detected objects and to establish their degree of danger. The visually impaired user is informed by the presence of different obstacles in his/her near vicinity through a set of acoustic signals.To demonstrate the reliability and robustness of the proposed architecture we have performed an extensive experimental evaluation with actual VI people. First, we have provided an objective evaluation of the obstacle detection and classification modules, while in the second part we have focused our attention in introducing a subjective evaluation of our system and we have presented the VI people degree of satisfaction and comments after using our prototype. The subjects found our system friendly, lightweight, wearable and non-intrusive, satisfying both the hands-free and ears-free requirements. Moreover, the computational time is reduced and the warning messages are transmitted fast enough so that the VI user can walk normally.For future work we plan to extend our framework with additional functionalities such as: providing navigational information so that a VI user can reach a desired destination, the development of a personal shopping assistant, or an additional face recognition capability to allow the identification of familiar faces. Moreover, it is necessary to derive a deeper level of understanding of the surrounding scene in order to provide additional cognition over the environment.


Satellite optical images and altimetry data are two major data sources used in Antarctic research. The integration use of these two datasets is expected to provide more accurate and higher quality products, during which data registration is the first issue that needs to be solved. This paper presents an alternative approach for the registration of high-resolution satellite optical images and ICESat  laser altimetry data. Due to the sparse distribution characteristic of the ICESat laser point data, it is difficult and even impossible to find same-type conjugate features between ICESat data and satellite optical images. The method is implemented in a direct way to correct the point-to-line inconsistency in image space through 2D transformation between the projected terrain feature points and the corresponding 2D image lines, which is simpler than discrepancy correction in object space that requires stereo images for 3D model construction, and easier than the indirect way of image orientation correction via photogrammetric bundle adjustment. The correction parameters are further incorporated into imaging model through RPCs  generation/regeneration for the convenience of photogrammetric applications. The experimental results by using the ASTER  images and ZY-3 (Ziyuan-3 satellite) images for registration with ICESat data showed that sub-pixel level registration accuracies were achieved after registration, which have validated the feasibility and effectiveness of the presented approach. Remote sensing observation of the Antarctic ice sheets, compared to traditional in situ measurements, is an important and relatively efficient method to acquire the spatial information of the ice-sheet topography as well as its change, and provides essential information for the study of global climate change and sea level rise ,2,3,4. SOptical remote sensing is another major data source used in the Antarctic research, which can provide high-resolution images with rich texture. The configuration of the along-track stereo cameras capable of recording near-simultaneous stereo images, can substantially improve the correlation between the stereo-pairs in extreme and changing environment . Multi-tSatellite optical imagery has an advantage in texture description, and the accuracy of horizontal positioning is higher than that of height determination. With regard to radar and laser altimetry, however, the accuracy in height is higher than that in horizontal plane, which is complementary to optical images. Therefore, the integration of these two different types of datasets is expected to produce products with higher accuracy. However, due to the difference in platform and data acquisition mechanisms, there always exists geometric inconsistency between the two different types of datasets, which needs to be eliminated through registration for further integration processing ,15.There is lots of published research regarding the registration of optical images and point cloud data ,36,37,38The fusion of ASTER stereo images and ICESat altimetry data was investigated in  to produThis paper presents an alternative simplified point-to-line feature based method in image space for the registration of satellite optical images and ICESat altimetry point data. It is based on direct discrepancy correction in image space through 2D transformation between the projected terrain feature points and the corresponding 2D image feature lines. The method can work for mono images and requires no conjugate points, which facilitates the registration of sparse points and the optical images. Following the detailed methodology of the approach, comprehensive experiments using the ASTER images and ZY-3 images for registration with ICESat altimetry data were conducted and the registration results are demonstrated and discussed, leading to the conclusion in the final section.Because the ICESat laser points are sparse, it is difficult and even impossible to identify the individual conjugate points from both the ICESat data and the optical images. However, both datasets contain terrain information, thus it is possible to obtain terrain features from both of them . TerrainIt should be noted that the ground features, especially those ice-rock boundary lines, may have moved due to the glacier change between the ICESAT measurements and the image acquisitions. In order to minimize the registration error introduced by possible glacier change between the acquisition times of different datasets, we chose the ICESat observations which were obtained close to the time or in the same season of the satellite image acquisition for the extraction of conjugate terrain features. In addition, a good distribution of the terrain features in the full image extent contributes to a robust and higher registration accuracy.The registration of ICESat laser points and optical images is based on the constraint that the projected terrain feature points extracted from ICESat data should coincide with the corresponding 2D feature lines measured from optical images. The 3D terrain feature points are first projected into image space via the vendor-provided imaging model—e.g., the RPCs provided along with the images. Due to the platform diversity and the errors in satellite orbit and attitude determination as well as inaccuracy of sensor calibration, there exist discrepancies between the projected terrain feature points and the measured feature lines in image space.Generally, there are two ways to eliminate the geometric discrepancies. One is to correct the resulting discrepancies directly in image space or object space, which can be referred to as the direct method. The other is to correct the imaging model parameters through photogrammetric bundle adjustment to achieve geometric consistency between the two types of datasets, which can be referred to as the indirect method. In the latter method, the parameters and observations which cause the discrepancies are adjusted, including satellite attitude and orbit observations (referred to as exterior orientation) and sensor calibration parameters (referred to as interior orientation). The indirect method requires an initial rigorous sensor model and is of high computational complexity. It is impracticable if only the RPCs are available. The direct method, on the other hand, is simpler for comprehension and easier for computation, and is applicable for both a rigorous sensor model and RPC model. According to the space where the discrepancies considered and corrected, there are two schemes for the direct method. One is to correct the discrepancies in image space and the other is in object space, the latter one was adopted in the research in . The objith projected terrain feature point, j = 0,1,2) are the polynomial transformation parameters. According to the selection of the parameters, we have different transformation models as follows.(a)(b)(c)(d)The discrepancies between the projected terrain feature points and the measured feature lines in image space can be modeled and compensated by polynomial transformation with constraint of minimizing the distance between points and corresponding lines. Let us say that Generally, higher order polynomials are not necessary and the affine model is efficient enough to model and compensate the discrepancies ,47.For each pair of matched features, we have distance observation equation:In Equation (2), n points, and k’s are the affine model parameters, the constructions of A, X and L are as follow:For n pairs of matched features, the distance observation equations can be written in the form of:The least-square solution of Equation (3) is:t is the number of transformation parameters) will be abandoned [In order to remove the possible outliers for a robust estimation, 3-sigma m × n × k  virtual object points. The dimension of the lattice should cover the range of the 3D terrain surface within the full extent of the image. To achieve a robust and accurate solution, the number of elevation layers should be greater than three, and the total number of object points should be adequate enough  considering 78 to-be-solved RPC parameters.Establishment of a three-dimensional (3D) lattice in ground space. The lattice contains (2)Determination of the corresponding image points with the bias being corrected. The virtual object points are projected into image space by using the original imaging model and are corrected by use of the image offset model parameters obtained in (3)RFM fitting. The new bias-free RPC parameters are then solved using the corresponding image and object grid points by least-square adjustment, during which the original RPCs (if available) serve as initial values.Compared with the rigorous sensor models, the RFM  is more popular and supported by most satellite image vendors and photogrammetric software. Therefore, bias correction can be incorporated into the photogrammetric model by generation/regeneration of a new set of bias-free RPCs. There a lot of studies have been published on RFM-based geometric performance evaluation and bias correction for various high-resolution satellite images ,51,52,53http://glovis.usgs.gov/). The images were level 1A products in HDF  format, with nominal resolution of 15 m and geolocation accuracy of 72 m [The satellite optical images used for experiments include ASTER VNIR (Visible and Near Infrared Radiometer) images and Chinese ZY-3 satellite images. The ASTER images were acquired from the Earth Resources Observation and Science Center (EROS) of USGS  . It was acquired between 2003 and 2009 by the onboard Geoscience Laser Altimeter System (GLAS). The altimeter samples 50–70 m diameter footprints every ~172 m along the track, with an elevation accuracy of about 0.15 m, and a pointing knowledge accuracy of ~2 arcsec, equivalent to 6 m horizontal geolocation error ,57. ConsThe matching primitives, including the terrain feature points and the corresponding feature lines, were extracted from the ICESat altimetry data and the optical images respectively following the principle described in By using the extracted control features, the transformation model parameters can be determined for each image by least squares adjustment as introduced in As discussed in Due to the difference in platform and systematic errors inherent in sensor calibration and orientation observations, geometric inconsistency exists between two different types of datasets, even between datasets obtained at different times from the same sensor, for which data registration is an essential procedure to eliminate the geometric inconsistency for integration use. Generally, translation could compensate a majority part of the inconsistency, which is corresponding to the discrepancies induced by constant sensor calibration errors and constant orbit and attitude measurement errors. However, there may also exist additional discrepancies after translation compensation, which could be attributed to the drift and/or higher errors with orbit and attitude measurements ,49. TherWith the geometric inconsistency being corrected by the transformation model in image space, the registration of the two datasets is achieved. However, the use of additional transformation model is not convenient in application such as DEM generation. Therefore, the transformation parameters are incorporated into the imaging model through RPCs generation/regeneration. The results of DEM discrepancy analysis along ICESat tracks conducted in A similar terrain-feature-based method was presented to register the historical aerial images to ICESat data in . It was Due to the sparse distribution of ICESat data, it is difficult and even impossible to find individual conjugate points between ICESat data and optical images, an alternative simplified registration method based on point-to-line discrepancy correction in image space is presented for the registration of the two types of datasets. Compared with the indirect way of inconsistency elimination by orientation parameter correction through photogrammetric bundle adjustment, the direct discrepancy correction method is simpler for comprehension and easier for computation. There are also two schemes for direct discrepancy correction according to the space in which it is implemented, among which discrepancy correction in image space is simpler and more generic than that in object space and can work for individual images, whether they are stereo or mono. Moreover, it is suggested that the correction result be incorporated into imaging model via RPCs generation/regeneration for the convenience of applications.The experimental results of using the ASTER images and ZY-3 images for registration with ICESat data show that sub-pixel level accuracies were achieved after registration. By comparing the DEMs derived from the stereo optical images and the ICESat data through elevation profile analysis along ICESat tracks, the results show a significantly higher consistency by using the regenerated RPCs with the correction model being incorporated after registration than using the original imaging models.At the current stage, the matching features were extracted interactively. In further research, more effort could be paid to the automation of feature extraction and correspondences matching to improve the efficiency for practical applications.


Ethical approval was obtained from the Ethical Commission of the Ghent University Hospital. Outcomes included objective data on the interventions ); as well as subjective experiences . There was a special focus on communication aspects within the introduction of this process. In total, 52 patients were included in the study, taking a median of 10 drugs (IQR 8–12). Upon admission, 122 drug discrepancies were detected. During medication review, 254 PIPs were detected and discussed, leading to an improvement in the appropriateness of medication use. The satisfaction of community pharmacists concerning additional communication and the satisfaction of the patients after counselling at discharge were positive. However, several barriers were encountered, such as the time-consuming process to gather necessary information from different sources, the non-continuity of the service due to the lack of trained personnel or the lack of safe, electronic platforms to share information. The communicative and non-communicative successes and hurdles encountered during this project need to be addressed in order to improve the full medication review process and to strengthen the role of the clinical pharmacist.For the majority of Belgian hospitals, a pharmacist-led full medication review process is not standard care and, therefore, challenging to introduce. With this study, we aimed to evaluate the successes and barriers of the implementation of a pharmacist-led full medication review process in the geriatric ward at a local Belgian hospital. To this end, we carried out an interventional study, performing a full medication review on older patients (≥70 years) with polypharmacy (≥5 drugs) who had an unplanned admission to the geriatric ward. The process consisted of 3 steps: (1) medication reconciliation upon admission; (2) medication review using an explicit reviewing tool (STOPP/START criteria or GheOP Transitions of care are defined as the movement of a patient from one healthcare provider or setting to another . SubsequA lack of communication between caregivers during these transfers of care has been shown to be linked to poor patient outcome  as well 3S) tool [By performing a medication review, potentially inappropriate medication (PIM) use can be detected. PIMs consist of over-, under-, and misprescribing of drugs ,11,12. I3S) tool . The complete process of medication reconciliation and review is a good example of a clinical pharmacy service, as it can be performed at an unplanned hospital admission. Clinical pharmacy services, in general, aim to provide patient care that optimizes medication therapy and promotes health, well-being, and disease prevention . HoweverTo support all Belgian hospitals to introduce or expand clinical pharmacy services, the government implemented supplementary financing  per 200 beds with a maximum of 2 FTE) . MoreoveThe new resources provided by the government has enabled the hospital described in this study to supplement the team with a clinical pharmacist (0.5 FTE). This study was conducted to explore the possibilities, the difficulties, and the stakeholders’ views on implementing a full medication review process in the hospital setting. We also aim to provide an overview of the communicative contacts during the process. This manuscript reports on the (communicative) successes and barriers that were perceived during a prospective interventional study that was carried out from March to September 2016. Patients were eligible if they (1) had an unplanned admission to the geriatric ward; (2) were at least 70 years old; (3) took at least five drugs chronically at the time of admission; (4) were not hospitalized in the preceding 3 months; and (5) did not have any form of cognitive impairment  <21/30 or documented confusion as determined by the clinical judgement of the physician or nurse). The study received ethical approval from the Ethical Commission of Ghent University Hospital. All patients provided an informed consent. The study was carried out in the General Hospital in Oudenaarde (Belgium), a local hospital with 235 beds and 3 hospital pharmacists. Prior to the study, a number of changes were introduced to enable the minimal conditions to perform clinical pharmacy. A vision on clinical pharmacy was outlined in cooperation with the Pharmacy and Therapeutics Committee. This included access to the full electronic patient file, granted to the pharmacists after approval by the medical board. In 2016, the main focus was shifted to medication reconciliation at the time of admission and discharge. Before the intervention described below, a hospital-wide basic protocol to improve medication reconciliation was developed and implemented in cooperation with the hospital pharmacy.The intervention consisted of an extended medication reconciliation at the time of admission and a medication review by the clinical pharmacist. Both were completed within 72 h of admission. Additionally, a medication reconciliation was performed at the time of discharge. A summary of the methods of communication is shown in For patients included in the study, the clinical pharmacist obtained the best possible medication history, starting with the initial medication list that was used by the ward upon admission. This initial list was composed by the treating physician with the support of a nurse, following the standard hospital procedure, making use of one or more extra sources including (1) the electronic prescription file if the patient was previously admitted; (2) the shared medical record from the collaborative care platform; (3) the referral letter of the GP, if applicable, and (4) the patient interview at admission. Subsequently, the full electronic patient file was consulted by the clinical pharmacist and discrepancies were noted. A structured interview with the patient or caregiver was performed in order to clarify the current drug use. For each drug included in the medication list, the current use, dosage, and time of administration were checked. Additionally, the limited questions list—which was proven they could reveal omissions —and quesThe treating physician was informed of every discrepancy between the initial medication list and the best possible medication list. Based on the current status of the patient, the treating physician adjusted the electronic prescription in order to provide optimal therapy. The detected discrepancies were classified as follows: (a) omission on initial list; (b) patient does not take this drug; (c) wrong dose; (d) wrong modality of administration (route/time of administration); (e) wrong formulation; (f) relevant information on duration of therapy; and (g) allergy previously not registered in file. Additionally, the contacted community pharmacists were questioned about (1) the presence of a medication list; and (2) their opinion on this inter-professional communication between pharmacists in primary and secondary care.3S tool [For the medication review process, patients were allocated to three different groups based on ward and treating physician. The first group consisted of patients treated by a geriatrician on one ward and these patients’ medication lists were screened with the STOPP/START criteria ,17. The 3S tool . PatientIn order to estimate the impact of the screening, the adapted MAI (aMAI) ,16 was cUpon discharge, the clinical pharmacist reviewed the discharge medication list by comparing it with the medications upon admission, changes during hospital stay, and instructions mentioned in the discharge letter. Unintended discrepancies between the different sources were discussed with the geriatrician and incorrect sources were adjusted in order to avoid confusion in the future. Immediately before discharge, the clinical pharmacists discussed the discharge medication with the patient or caregiver. Changes were emphasized, such as starting, changing, or ceasing medication. The necessary counselling was provided for new drugs, such as administration instructions , alarm symptoms of frequent adverse drug reactions, or possible important interactions with food or other drugs. At the end of this counselling, the patients or caregivers were asked if this extensive information session was helpful in understanding the discharge medication list. Moreover, to evaluate the effect on transfer between settings, a pharmaceutical discharge letter was sent to half of the general physicians and community pharmacies of the included patients. This letter contained the discharge medication list, as well as details on the changes compared to the prior medication list. During a follow-up phone call to the receiving general physicians (GP) and community pharmacists, the added value of the pharmaceutical discharge letter was discussed. Three questions were asked: (1) Do you think this letter is useful? (2) What actions do you undertake upon receiving the letter ? (3) Do you have suggestions to improve this clinical pharmacy activity? A special focus was placed on the different needs for communication within every step of the process. Upon initiation of the pilot project, all internal stakeholders  were asked to gather their impressions and experiences (both positive and negative) during the process. These experiences were shared during the other contacts with the clinical pharmacist. Moreover, the professionals were asked to provide ideas on how the performed clinical pharmacy activities could be improved. A short report was written by the clinical pharmacist after every contact that contained relevant information on the process and the impressions of the people involved. This project aimed to identify the successes and barriers encountered during the implementation of a full medication review process as an example of a clinical pharmacy service in a hospital. The primary outcomes were number and type of medication discrepancies at admission and the improvement of prescribing appropriateness. All other outcomes were considered secondary. Both were evaluated and analyzed by the pharmacist in a descriptive way. Descriptive statistics are displayed as counts with percentages, median with interquartile ranges (IQR), and means with standard deviations (SD) as appropriate. Information about experiences is reported qualitatively.The combination of the results of the interventions and the experience of all participants, both patients and healthcare workers, resulted in a summarizing box for each step of the project, divided into three parts: (1) successes; (2) barriers; and (3) recommendations for the different involved partners. n = 20); (2) taking <5 drugs at home (n = 38); (3) admission to a geriatric ward less than 3 months ago (n = 40); (4) cognitive impairment (n = 94), or (5) absence of the clinical pharmacist or no consent (n = 17), and received standard care. In total, 52 out of 261 (20%) admitted patients were included in this study. The other 80% of patients were excluded based on the first exclusion criteria following this cascade: (1) age < 70 ). In total, 122 discrepancies  were detected; 58 (11%) were obtained or altered after the structured interview with the patient or caregiver (median of 1 per patient (IQR 0–2)) and 64 (12%) after consultation with the community pharmacist (median of 1 per patient (IQR 0–2)). For 10 patients (19%), no community pharmacy could, however, be consulted . For some patients, the general physician (GP) was contacted when relevant information was missing. These communications often required multiple attempts due to inaccessibility of the GP or patient file while the GP was performing home visits. The types of discrepancies are depicted in n = 20, some patients visited the same community pharmacy) was positively surprised with the request for additional information and considered this contact positive. For only two patients, the community pharmacist said a medication list was available. One pharmacist mentioned that she asks patients with complex schemes whether they desire one and often (9/10) patients say they do not want or need one. Every contacted community pharmacist  and 15.2 (±4.9) min, respectively. A total of 254 PIMs were detected, resulting in 195 therapeutic recommendations for the treating geriatrician after clinical judging by the pharmacist (see n =20) or STOPP/START criteria (n = 17), respectively. In the control group (n = 9), this improvement was seen in only 55.6% of patients  of 23.5 (±10.5) min. The subsequent mean (±SD) time needed to perform the medication review using the GheOPcist see . Thirteepatients . Due to A total of 46 of the included patients were discharged within the study period. Six patients were lost in follow-up due to death during admission or discharge beyond the study period. Forty-one medication lists were evaluated by the pharmacist upon discharge, as 5 patients were discharged during the absence of the participating pharmacist. Overall, there was a good agreement between the medication list and the information in the discharge letter from the physician. The most important detected non-agreements were the omission of warfarin on a scheme and the wrong dose of pantoprazole. Other interventions by the pharmacist were the substitution to the brand or generic drug used at home from a different brand or generic drug during hospitalization or the adjustment of modalities of administration . Every patient with whom the medication list was discussed evaluated this intervention positively.For 24 patients, a pharmaceutical discharge letter was sent to the general physician and community pharmacist. Seventeen of the 18 pharmacists thought the information in this letter was useful and saved the information in the pharmacy software or patient file. Only 7 of the 16 general physicians were positive. Main reasons to not support this were (a) information is already available in the discharge letter; (b) unnecessary with the anticipation of an upcoming shared electronic patient file; (c) additional administrative burden (another letter to read and file).We performed a prospective interventional study, implementing a full clinical pharmacy service in a local Belgian hospital. The service consisted of medication reconciliation at the time of admission, a medication review, and a medication reconciliation at the time of discharge. Fifty-two patients were included. A discrepancy was detected for about one in four drugs. Medication review with a validated tool improved medication appropriateness according to the MAI score. A discharge consultation with the clinical pharmacist was positively evaluated by the patients. The results show a positive influence of a pharmacist-led medication reconciliation upon admission, which is in line with many previous studies ,8,25. ItMany of the detected discrepancies led to changes in the prescriptions during the first days of admission and thus decreased the risk of therapeutic failure or medication errors. Some of the omissions concerning (non-)prescription drugs provided valuable information on additional symptoms experienced by the patient, as these symptoms could be signs for adverse drug reactions . Additionally, the use of, for example, calcium supplements at home is relevant due to the possible influence on the absorption of several drugs. In contrast, some of the discrepancies were not taken into account by the physician in his evaluation of the prescriptions, such as the omission of multivitamins or homeopathic preparations. As communication between the pharmacist and geriatrician was performed by telephone call or by e-mail, the only trace was the adjustments by the geriatrician in the electronic prescriptions. In order to improve traceability of the activities and interventions, as well as to provide all known information to every healthcare provider of the patient, future projects would benefit from registration in the electronic patient file. Ideally, this should be done in a section specifically designated for pharmacotherapeutic information.The request for information on the medication dispensing history was positively experienced by the community pharmacists. Community pharmacists have a strong therapeutic relationship with patients and have access to the full dispensing records. Community pharmacists should use this important qualification in, for example, providing medication lists to patients. They should also inform patients on the helpfulness of an updated medication list in (often unexpected) transitions of care. This key role of the community pharmacist has been recently confirmed by the approval of a multiannual framework in Belgium . PatientAn important barrier to implement a thorough medication reconciliation as standard care is the time to perform this task. Obtaining the best possible medication history through the above described procedure requires staff who are sufficiently trained to perform this task, possibly a pharmacy technician. Moreover, it is a time-consuming process and staffing is not equal 24/7, making it difficult to provide this service on a regular basis. However, the process of medication reconciliation could be facilitated through a centralized  patient file, which creates the possibility to share information between healthcare providers and settings. The Virtual Integrated Drug Information System (VIDIS) platform is being developed by the Belgian government for this purpose but is not yet implemented in every hospital or pharmacy and is, therefore, not yet usable. Until the implementation of this platform, the need for an extensive process of medication reconciliation remains. A summary of the successes, barriers and recommendations concerning medication reconciliation upon admission can be found in The current software system used in the hospital is not sufficiently integrated to facilitate the creation of a useful pharmaceutical file consisting of crucial patient diagnostic, laboratory, and prescribing (history) information. The need to consult many different sources increases the risk of missing relevant items. Moreover, gathering the information takes a lot of time and decreases the number of patients for whom a medication review can be executed. A clear, uncomplicated patient file is again a must-have in order to perform medication reviews in a timely manner. This integrated patient file should be available for every professional with a therapeutic relationship with the patient. Also, physicians and other healthcare workers should be motivated to keep the file up to date and to make sure that everyone who is involved in the treatment of the patient is sufficiently informed.The time needed to perform a medication review and to formulate the recommendation was comparable for both tools. This shows that when a clear patient file is present, medication review can be performed within a reasonable amount of time. The adjustment of about 20% of the potential inappropriate prescriptions depicts the potential role a clinical pharmacist can play in the optimization of the medication list.In most cases, the geriatrician read the internal e-mail containing the recommendations; however, some were missed and recommendations were only evaluated during the weekly meeting. In the case of acceptance of the recommendation, this could lead to a delay in therapeutic change of up to 7 days. To avoid these unnecessary and potentially harmful delays, adjustments in communication strategy are necessary. For example, when recommendations are communicated through e-mail, reading confirmations should be asked for. Moreover, everything should be registered in the patient file, readily available for the physician, and a system should be set up to notify the physician that recommendations are present. A more intense follow-up with a reminder for the physician after a certain amount of time could also be beneficial. Another way to improve communication is the presence of the clinical pharmacist during multidisciplinary meetings. As such, medication-related issues could be addressed on a regular basis. The pharmacists can also be present on the ward or participate in ward rounds in order to provide immediate pharmacological support, a strategy that has been proven to be successful ,28,29. TDischarge is another important transfer of care that requires specific attention. The patient or caregiver should be informed on the correct therapeutic plan to follow after the hospitalization, which starts with a correct medication list. Evaluation of the scheme by the clinical pharmacist showed that the majority of the schemes were concordant with the actual therapeutic plan of the physician, as noted in the patient file and the discharge letter. However, the detection of some errors confirmed the usefulness of a review, as in rare cases important drugs were omitted or the dosage was incorrect. Additionally, the frequent need to substitute some drugs to the brand the patient was using prior to hospitalization is an important intervention as this could lead to confusion or even double intake of the same drug. The preparation of the medication list prior to discharge is a condition to be able to perform this clinical pharmacy activity. During the study, the participating geriatrician always provided the scheme on time. The software supporting this activity was, however, evaluated to not be user friendly, keeping other physicians from preparing a medication list. The participation of only one clinical pharmacist in the project caused the service of the evaluation of the discharge medication list to not be continuous.Due to the lack of a safe electronic system used by every healthcare provider to share patient information, the pharmaceutical discharge letters were sent by conventional mail. Community pharmacists reacted overall very positively to the pharmaceutical discharge letter. They believed they could provide better pharmaceutical care for their patient as they could inform the patient on the changes. The therapeutic plan was also clearer for the pharmacist him/herself, which made them more confident they were dispensing the correct drugs. General physicians’ opinions on the pharmaceutical discharge letter were divided. The reason to support the use of this letter was the elimination of the need to search the entire discharge letter for scattered information on pharmacotherapeutic information. A summary of the experiences and recommendations on medication reconciliation upon discharge is provided in To our knowledge, this is the first study to provide an overview of the communicative contacts during a full medication review process by a clinical pharmacist. The successes and barriers were analyzed from both a quantitative and qualitative perspective, taking into account the opinions of every involved participator in the process. Additionally, the large amount of information and experiences were used to formulate recommendations, which could support the short- or long-term improvement of the implementation or expansion of a full medication review process. The recommendations described in the study could apply to different patient populations, as medication reconciliation or medication review are important processes for every patient undergoing a transition of care, taking multiple drugs, or with a complex pathology. Moreover, the results from the study could be beneficial for other types of institutions, such as nursing homes, as they also often struggle with the same barriers. Nevertheless, this study has some limitations. The small number of included patients precluded the statistical comparison of the groups, resulting in only descriptive statistics. This project shows there is a need to include patients with cognitive impairment in studies evaluating the effect of medication screening tools, as they represent a major part of geriatric patients requiring hospitalization. Moreover, the positive effect of screening on medication appropriateness, as reflected in improvement of outcome parameters such as MAI score, cannot be extrapolated to a positive effect on other outcomes, such as drug reactions, drug-related hospital admissions, healthcare use, patient reported outcomes, and mortality ,31. ThesAnother limitation was the participation of only one clinical pharmacist and one geriatrician, compromising the continuity of this clinical pharmaceutical service. For the pharmacists, the problem is dual: (1) a shortage of staff to provide continuous service; and (2) the lack of training of the hospital pharmacists to perform this task. Shortage of staff is, however, being partially addressed by the Belgian government through structural financing. Nevertheless, the level of knowledge of this subject in Belgian hospital pharmacists should improve in the upcoming years as the training of hospital pharmacists has been prolonged (from 1 to 3 years), with additional focus on clinical pharmacy , and hospital pharmacists could further specialize in clinical pharmacy by following additional courses ,32. InfoThe experiences of the involved healthcare workers were collected by the same clinical pharmacist who was involved in the execution of the clinical pharmacy service, in reports of the often verbal inter-professional contacts. As this could possibly introduce bias in the obtained results, future studies could benefit from written questions or independent interviewers. Additionally, a formal analysis with focus groups is needed; however, this was beyond the scope of this implementation study.Clinical pharmacy services can only be effective if the treating physician uses the results to optimize patient treatment. Therefore, the results from the pilot project will be reported to the medical staff of the hospital in order to create awareness on the potential role of clinical pharmacy for their patients and to stimulate their enthusiasm. After performing and evaluating this pilot project, we are ready to introduce front-office clinical pharmacy at our local hospital. Every project should be preceded by adequate training of the involved pharmacists and other healthcare workers, as well as by the definition of a clear communication plan. The differences in expectations on inter-professional communication between pharmacists and GPs have been recently described for the German situation by Weissenborn et al. . A similAs demonstrated in the analysis of our pilot project, further extension of clinical pharmacy, in general, will need optimization on different levels. Healthcare providers should receive intensive training and should recognize each others’ roles in the optimization of medication and patient safety. Interdisciplinary and seamless healthcare can be facilitated by the creation of structured patient files that are easily and securely accessible by every professional and with the possibility to communicate. This should be developed both locally  and externally through the VIDIS platform currently under development. The introduction of clinical pharmacy should be considered in every hospital as this has a significant impact on patient safety and patient care. The importance of a prominent role of the clinical pharmacist in the optimization of drug use in older people through the implementation of a full medication review process has been shown in this study. However, opposite every success, several barriers were detected or perceived, showing the difficulties in setting up a structure to be able to perform clinical pharmaceutical activities. Clear inter-professional communication has been shown to be essential during every step in order to obtain positive results and to facilitate the process. The identification and analysis of the communicative and non-communicative successes and hurdles encountered during this project need to be addressed in order to improve the full medication review process.