Smart buildings benefit from IEEE 802.15.4e time slotted channel hopping (TSCH) medium access for creating reliable and power aware wireless sensor and actuator networks (WSANs). As in these networks, sensors are supposed to communicate to each other and with actuators, IPv6 multicast forwarding is seen as a valuable means to reduce traffic. A promising approach to multicast, based on the Routing Protocol for Low Power and Lossy Networks (RPL) is Bidirectional Multicast RPL Forwarding (BMRF). This paper aimed to analyze the performance of BMRF over TSCH. The authors investigated how an adequate TSCH scheduler can help to achieve a requested quality of service (QoS). A theoretical model for the delay and energy consumption of BMRF over TSCH is presented. Next, BMRF’s link layer (LL) unicast and LL broadcast forwarding modes were analyzed on restricted and realistic topologies. On topologies with increased interference, BMRF’s LL broadcast on top of TSCH causes high energy consumption, mainly because of the amount of energy needed to run the schedule, but it significantly improves packet delivery ratio and delay compared to ContikiMAC under the same conditions. In most cases, the LL unicast was found to outperform the LL broadcast, but the latter can be beneficial to certain applications, especially those sensitive to delays. Wireless sensor and actuator networks (WSANs) comprise constrained devices equipped with sensors and/or actuators. These devices , which can use radio waves to communicate, are exposed to noise and interference, and they are often powered by batteries. They have limited memory and processing capabilities, and they dispose of limited bandwidth, directly reflected in their networking performance. Therefore, WSANs should benefit from new protocols and applications customized for their needs, especially regarding energy and bandwidth. At the medium access control (MAC) level, protocols are responsible for minimizing collisions and maintaining energy consumption at a low level by putting the nodes’ radios to sleep as much as possible with so called radio duty cycling (RDC) protocols. Contiki 3.x and the next generation (NG) Contiki-NG are operating systems (OSs) developed for sensors and actuators. They have a different protocol at the RDC/MAC level—ContikiMAC  for ContIn smart homes, point-to-multipoint (P2MP) communication is extensively used. Examples of this type of communication are measurements transmitted by a temperature sensor to interested devices, such as the heating control, the user’s smartphone, a wall display, or a command transmitted by the central system to several actuators controlling the light fixtures in the house. Internet protocol (IP) multicast improves the efficiency of P2MP communication. When using the Internet Protocol version 6 (IPv6) unicast, the source sends an individual packet to each destination, whereas with IPv6 multicast the source sends only one packet with an IPv6 multicast address that the routers duplicate if needed in order to forward it to the destinations that subscribed to that multicast address. The use of multicast addresses further enhances the network’s performance by avoiding the need to keep a list of interested destination IP addresses at the device level.In this paper, IPv6-based multicast over TSCH and ContikiMAC empowered WSANs is discussed. The investigated protocol, bidirectional multicast RPL forwarding (BMRF), is based on the Routing Protocol for Low Power and Lossy Networks (RPL). It supports both link layer (LL) unicast and LL broadcast forwarding to send IPv6 multicast packets downwards in the RPL tree to interested nodes (nodes that are part of the multicast group). BMRF can adapt its mode of operation at the link layer in each node to help to achieve the requested quality of service (QoS). Furthermore, a theoretical model for the delay and energy consumption of BMRF over TSCH with a given scheduling algorithm is presented. The focus is on delay and energy consumption. Along with the development of a theoretical model, simulations considering different scenarios with less or more interference between nodes are provided.The purpose of this paper was to investigate under which conditions multicast offers the most feasible and adequate solution. In order to do this, both LL unicast and LL broadcast forwarding in BMRF over TSCH- and ContikiMAC-enabled networks were thoroughly examined.The paper is organized as follows. Multicast is an advantageous communication means in WSANs. An important number of papers describe the implementation of new multicast protocols. At least three other recent papers have analyzed BMRF, the network layer protocol we used in this research.In , the autIn , the autNone of the above-mentioned protocols used the threshold in BMRF that allowed for the nodes to switch from the LL unicast mode to the LL broadcast mode, depending on the node’s number of interested children and the network’s requirements. None of these papers discussed the interaction between BMRF and the MAC layer.The authors of  proposedA new RPL-based multicast routing mechanism for wireless sensor networks was proposed in . The autAs shown in the paper proposing BMRF , the RDCAs for a comparison between TSCH with 6TiSCH minimal configuration and ContikiMAC, the authors of  comparedTo the best of our knowledge, no thorough analysis yet exists for BMRF over TSCH, and no comparison between LL unicast and LL broadcast forwarding has been provided, which was the goal of this paper.The protocols under study in this paper were IPv6-compatible RPL, BMRF, TSCH, and ContikiMAC. RPL , a distaThe DODAG is constructed with the help of control messages. The Internet Control Message Protocol for IPv6 (ICMPv6) information messages that play a role in building the DODAG are: DODAG Information Solicitation (DIS) messages, DODAG Information Object (DIO) messages, destination advertisement object (DAO) messages, and destination advertisement object acknowledgment (DAO-ACK) messages. Three modes of operation: unicast, broadcast, and mixed; thus, in unicast mode, all nodes use LL unicast to forward packets, and in broadcast mode, they use LL broadcast. In the mixed mode, each node decides whether to forward the IPv6 multicast packet using either LL unicast or LL broadcast according to a threshold based on its number of interested children (children that are group members or must forward the IPv6 multicast packets).Bidirectionality: multicast packets can be forwarded both upwards and downwards, allowing for the source of multicast traffic to be located in whichever node of the network.Duplicate avoidance.In order delivery of IPv6 multicast packets.Multi-sourcing: more than one node can, at some point in time, send multicast packets to the same multicast destination address .Dynamic group registration: BMRF follows the subscription process described in RPL and implements an adequate un-subscription process. Group registration is possible via RPL DAO messages. A child can unsubscribe from the multicast group by sending a special DAO message called the No-Path DAO. In this case, the parent removes the route to the child.BMRF  is speciBMRF manages to tackle some of the shortcomings of SMRF and RPL multicast, allowing for sources of multicast traffic to be located inside the networks and supporting dynamic group registration. This new protocol is configurable and, with optimal settings, can lead to fewer radio transmissions, less energy consumption, and a higher packet delivery ratio compared to SMRF .TSCH  structurA timeslot is long enough to accommodate the transmission of a frame, as well as the reception of its acknowledgement. This means that the length of a timeslot depends on the physical layer parameters in the network. Two coordinates define a timeslot from a slotframe, the time offset, and the channel offset. The time offset shows when the timeslot starts in a slotframe, and the channel offset determines the frequency channel in which the communication occurs.Through judicious cell assignment and channel hopping, TSCH is a reliable approach for low power networking. Cell assignment should be done in such a way to accommodate the predictable and/or event-driven data streams while avoiding collisions as much as possible. Power and throughput-efficient scheduling is challenging in highly dynamic networks. It is indeed less efficient when communication needs are not recurrent and cannot be predicted. A scheduler dictates a node whether to transmit, receive, or sleep within each timeslot. There are two main approaches for scheduling: centralized and decentralized. In the first approach, a central entity periodically collects data , computes a schedule and disseminates it back to the rest of the nodes. When every node locally computes its own requirements in terms of scheduling, exchanges information with neighbors, and negotiates specific slots, the scheduler is decentralized. Orchestra introduces autonomous scheduling. In this case, every node locally computes its schedule without exchanging information with neighbors since other protocols, e.g., RPL, collects the needed information beforehand.The Orchestra scheduler can be used in Contiki to build the schedule that dictates what a node should do during each timeslot. The rules that govern this schedule are expressed in terms of TSCH slotframes and slots with specific properties .There are four types of active slots in Orchestra: rendezvous or common-shared (CS), receiver-based shared (RBS), sender-based shared (SBS), and sender-based dedicated (SBD) slots . When usSome of the problems experienced with ContikiMAC are due to its single channel communication. Multichannel communication might offer a solution, and its potential can be exploited in wireless networks for simultaneous transmission on orthogonal channels. Multiple channel communication can allow for a more efficient use of the available bandwidth. Another mechanism employed by TSCH is channel hopping, which ensures that a timeslot switches frequency in a next slotframe. Channel hopping can reduce the influence of external interference .After thoroughly analyzing how Orchestra is implemented in Contiki-NG, we concluded that there is no rule that can accommodate only LL broadcast/multicast traffic. Since we intended to investigate whether LL broadcast/multicast was worth considering, we decided to implement a dedicated rule for this kind of traffic. It was observed that in the case of TSCH with default Orchestra settings, data packets sent via LL broadcast frequently collided with control packets used for RPL .Receive slots are not scheduled for broadcast traffic originating from children.With the help of an attribute, downward broadcast is differentiated, and the desired rule is triggered.The default operation of Orchestra was extended with a rule that accommodates only the data broadcast traffic. This new rule schedules a slotframe for LL multicast transmission containing sender-based shared slots. In other words, one broadcast-related transmitting slot and as many broadcast-related listening slots as the number of children are allocated for every node. This rule takes the particularities of the BMRF protocol into consideration. BMRF drops all broadcast messages not originating from its RPL parent to avoid receiving and forwarding the same messages several times, upstream BMRF traffic being always unicast. Therefore, we made the following alterations:After running simulations with BMRF in the broadcast mode with the Orchestra default broadcast rule and the above-mentioned rule, we observed that there was an improvement of the packet delivery ratio and delay for the newly implemented rule.Placement such that internal interference is kept at minimum. This was mainly used to be able to devise a simplified theoretical model for ContikiMAC and TSCH. It resulted in a RPL tree that would be favorable for ContikiMAC .Placement with interference only between nodes of the same rank in the RPL tree .Random placements that resulted in the RPL trees, as shown in The simulations and placements of the nodes in the network were designed to analyze the performance of BMRF with TSCH and ContikiMAC, as well as to investigate under which circumstances multicast was an efficient solution for WSANs. A 150 × 150 m square field was used to place the nodes. Each node had a transmission range of 50 m, equal to the interference range. Three types of node placement were chosen:In each of the mentioned cases, node 1 was selected to be the root for the RPL tree. The rest of the nodes were placed in such a way that after running RPL, the RPL trees in In For Subsequently, two topologies with random placements of nodes and increased internal interference were used to test the protocols in realistic scenarios. These topologies are shown in We defined configurations featuring the following characteristics: BMRF in one of the specific modes (unicast or broadcast) and TSCH or ContikiMAC as RDC/MAC protocols. Each of these configurations was uploaded on every described topology, and we ran 10 simulations for each case. The source of the multicast messages was the root of the RPL tree. It transmitted 100 multicast packets with a 4 byte payload at a rate of 0.2 packets per second in each case. We chose these values to avoid queueing delays. Since TSCH is the default protocol in the Contiki-NG OS  and ContAll the presented networks were built using the Zolertia Z1 mote in the Cooja simulator. The power consumption of the Zolertia Z1 radio could be found in its datasheet and equaled 56.4 mW (3V × 8.8 mA) for receiving, 52.2 mW (3V × 17.4 mA) for sending, and 0.0015 mW (3V × 0.5 µA) when it was in low power mode .Note that the default schedule of TSCH was adapted by adding a new rule dedicated to LL broadcast data traffic, as mentioned in The theoretical model intended to explain and give insight about the delay and energy consumption of a TSCH-enabled network. The theoretical model for ContikiMAC is presented in the UP is the unicast period expressed in timeslots. This leads to an average access delay for one hop of Delay TSCH unicast: chances of collisions when running a TSCH-enabled network are negligible because all motes are waking up at different times and on different channels. When using TSCH with Orchestra, the default setting for unicast includes RBS slots, so when a packet is received and needs to be forwarded, the mote must wait until the receiver’s unicast timeslot. This happens at earliest in one length of a timeslot (TSL) and the latest in one slotframe, which equals to Delay TSCH broadcast: When considering the default Orchestra rules, the delay for broadcast in TSCH at the first hop follows the same reasoning as for unicast: the sender must wait for the receivers to wake up. This is After the first hop, when the packet needs to be forwarded, the mote must wait for exactly one slotframe to be able to forward its packet, resulting in the average delay expressed below:The new rule allocates a listening timeslot for every child a parent has, resulting in a lower delay than the one of TSCH unicast. The packet delivery ratio (PDR) also increases since adding an entire slotframe only for data broadcast traffic greatly eliminates the probability for collisions between the RPL control packets and data broadcast traffic. The cost for these improvements is a rise in energy consumption.Energy: Energy consumption can be split in two main parts: the energy consumed by the traffic (sending and receiving packets) and the energy consumed by the RDC mechanism (when no packets are being sent):The energy consumed by the RDC mechanism is dependent on whether the time the radio is awake, as well as on the energy consumption of the radio:In real topologies, the clocks of the nodes deviate from the nominal frequency because of various parameters such as temperature, supply voltage, and manufacturing variances. The clock drift between nodes leads to problems in synchronizing at the timeslot level that can eventually cause communication failure. To compensate for these drifts, guard times are introduced at the beginning of the timeslots. For these time intervals, the receiver turns its radio on, before the scheduled transmission, in order to guarantee the reception of the packet see . The actIn a multi-hop network, the width (the number of nodes per hop) of the network has a direct impact on the clock drift, which increases with the width of the network . In the In this section, the results of the simulations with BMRF are interpreted for all the topologies in Firstly, we analyzed the results for PDR. In Looking again at Simulating LL broadcast on topologies with interference between same rank nodes resulted in comparable PDR performance for the two protocols b,d. IncrAnother parameter used to analyze the performance of TSCH and ContikiMAC and the two forwarding modes, LL unicast and LL broadcast, is the delay per hop. The results for the delay are illustrated in The results for topologies without interference and LL unicast, shown in However, for topologies with a random placement of nodes and LL unicast, ContikiMAC showed much larger delays c than TSFor the topologies where the same rank nodes interfered with each other , and forThe delay in BRMF over TSCH-enabled networks was lower for LL broadcast forwarding b compareThe last parameter used to compare LL unicast with LL broadcast forwarding, and the two protocols between them was the energy consumption per hop. The results for this performance parameter are shown in The normal operation of TSCH (TSCH synchronization messages and messages for operating the schedule) used most of the energy (compared to the sending/receiving of packets), resulting in an overall higher energy consumption. In contrast, for ContikiMAC, sending/receiving was inefficient when collisions appeared. This behaviour could also be seen in the theoretical model . AnotherFor the topologies with no interference or with interference only between same rank nodes, the energy consumption of BMRF on top of TSCH and in the broadcast mode was higher than the energy consumption for LL unicast forwarding Placing nodes randomly and creating a lot of interference between them resulted in an increased energy consumption when using ContikiMAC, whereas this had no effect when using TSCH; see In this paper, the influence of the MAC/RDC protocol on the performance of IPv6 multicast-enabled WSNs was studied. We analyzed the default RDC of Contiki 3.x (ContikiMAC) and the default RDC of Contiki-NG (TSCH), as well as how the link layer broadcast and unicast forwarding of the IPv6 multicast packets behaved in terms of PDR, delay, and energy consumption.In terms of PDR, for ContikiMAC and TSCH, LL unicast outperformed LL broadcast. This was mainly due to the LL unicast retransmission mechanism. We demonstrated that setting the number of retransmissions to zero notably decreased the PDR for the LL unicast. We also showed that on topologies with an increased level of interference, TSCH outperformed ContikiMAC regarding PDR.In terms of delay, for topologies with low interference, BMRF in the unicast mode with ContikiMAC outperformed BMRF in the unicast mode with TSCH since the time between two wakeups in ContikiMAC was lower than the time between available slots in TSCH. However, after increasing the interference between nodes, TSCH with our custom settings became the better protocol because most transmissions were done on different channels and, therefore, the level of interference in the network had no effect on the delay. In terms of delay, BMRF in the broadcast mode on top of TSCH as found to be the preferred solution for topologies with the random placement of nodes and a lot of interference.The energy needed to operate TSCH without sending packets was higher than the energy needed for ContikiMAC, since a slot in TSCH resulted in a much longer wake-up time than a clear channel assessment (CCA) from ContikiMAC. However, as found to be the preferred protocol when there is a lot of traffic, mainly because of the scheduling mechanism. This mechanism reduces interference, which saves energy.Future work on BMRF and how it operates on top of TSCH should consider the implementation of an entirely new scheduler. The scheduler should be efficient for both LL unicast and LL broadcast transmission. Moreover, it should adapt to highly dynamic and scalable networks. Clear guidelines and mathematical models should help determine the value for the threshold that enables the switch between LL unicast and LL broadcast in the case of the BMRF mixed mode.In general, we conclude that TSCH and multicast forwarding have a lot of potential but might need some more improvements to become the best solution available for all situations.


Understanding social interactions in livestock groups could improve management practices, but this can be difficult and time-consuming using traditional methods of live observations and video recordings. Sensor technologies and machine learning techniques could provide insight not previously possible. In this study, based on the animals’ location information acquired by a new cooperative wireless localisation system, unsupervised machine learning approaches were performed to identify the social structure of a small group of cattle yearlings ( Studying individual interactions in animal groups improves our understanding of social dynamics, networks, and hierarchies. The outcomes of such research applied to livestock groups can potentially improve livestock productivity, enhance management practices, and provide scientifically-validated recommendations for regulations and legislation of the farming industry . Cattle Behavioural analysis of animals is non-trivial. Accurately observing animal behaviour requires minimal human interference, and reliance on visual observations alone can be subjective and unreliable . The feaSocial networks and/or movement patterns in cattle have previously been determined using a range of methodologies, including live observations of proximity, leader animals, and scoring of social rank ,6,14, GPData generated from both proximity, GPS, and localisation devices can total thousands of registered datapoints and traditional methods of analysis may be insufficient for revealing potential patterns of group behaviour. Machine learning, in particular, unsupervised learning or clustering, has been applied to process the location data of animals, to study survival status, living habits, migration range and path of animals, to obtain knowledge of animal behaviour and social interactions ,26. ‘BigAlthough a local-position measurement system and positional loggers have been used for some research ,17, in mThus, the purpose of this research was to develop an efficient and accurate approach to study the social behaviour of cattle groups based on the novel application of a high-accuracy wireless tracking system and the latest machine learning technologies. Novel data analysis algorithms were developed and applied to the experimental data, which enabled the parameterising and visualising of the bonding between different animals, the influence of individual animals, and the strong repulsion (or avoidance) between individuals.The experiment was approved by the CSIRO FD McMaster Laboratory Chiswick Animal Ethics Committee prior to the start of the experimental period (ARA 18-06).The experiment took place at the Chiswick research site  of the Commonwealth Scientific and Industrial Research Organisation (CSIRO) in August 2018  closeness between any pair of animals in the experimental group, the ranks of their distance were used with reference to their respective distances to the rest of the group. At every time instance, the ranks were recorded, with these ranks indicating how closely two animals preferred to stay together and interact throughout the experimental period. Because the animals could roam throughout the test paddock, the physical distances could vary drastically, rendering them less meaningful and potentially biased or distorted, thus ranks were used.i was evaluated with respect to animal j  is an effective means of visualising the level of similarity between samples in a dataset . It has i with respect to j, A symmetric matrix  follows (2)B=−12K-value, K-means clustering randomly selects K points as the initial centroids and evaluates the distance from every sample to the K centroids one after another. Then, the centroids of the clusters are re-evaluated, and the clusters of the rest of the samples are updated accordingly [Inspection of  applied  to revea applied . Given tordingly . This rei with respect to j at timeslot t, as defined earlier. The matrix indicated the amount of variation or dispersion between the connection of animal pairs. Following the method described in The variation of the distance ranking of the animals throughout the experimental period was also further analysed to understand repulsion between some individuals. A new asymmetric square matrix, Agglomerative Hierarchical Clustering (AHC) is a bottom-up clustering method that can classify the samples based on pair-wise distances  . At the To apply the AHC algorithm, the affinities between the animals were first defined. This started by defining the outdegree which characterised the correlation from an animal to a cluster, and the indegree of an animal from a cluster that reflected the density near the animal .i, the average indegree from cluster i, and C. j and i defined earlier. If cluster i, both the indegree and outdegree were expected to be large. Therefore, the affinity between cluster i could be measured by the product of the indegree and outdegree [For any animal utdegree :(6)Ai→C=Likewise, the affinity between two clusters Based on the weighting coefficients in At the beginning of AHC, every animal formed a cluster of its own, and the indegree and outdegree of the cluster were Based on , the affremained , as descIn addition to the logical distances defined in The location estimations were generated at a fixed rate of 10 Hz during the experiments, which included the x- and y-coordinates and the associated timestamp. A total of 17,933 valid position records were obtained for each animal during one 25-min competitive feeding scenario. Dv in ) to a twThe same approach was applied to all timeslots within the captured dataset, and the statistics on the results are shown in K-means clustering analysis). This shows that animals 1 and 7 spent most of their time alone, totaling 6690 and 3842 timeslots respectively . This indicates that animals 1 and 7 were less connected to the others. It can be seen from This study applied novel wireless localisation devices to measure the geo-spatial locations across time of a group of 10 yearling cattle at fine spatial resolution during walking from the yards into a paddock area where a competitive feeding set-up was placed. The data points collected across the 25-min study period were analysed with two machine learning clustering algorithms both of which demonstrated that one animal was most influential or central within the group, that animals preferred to be in groups of 3 or 4 individuals, and that two individuals spent more time alone and were likely repelled by the central individual. This first application of these devices and analysis methods demonstrate the information that can be gained about social interactions and networks in animal groups from short observation periods.The wireless tracking system used in the experiment combined a range-based positioning system (WASP) and GPS receivers for improved accuracy. It was robust against measurement errors by exploiting all available measurements, including the pairwise range measurements between animals. To obtain positioning data with the highest quality for analyses, several repeated measurements were conducted on the animals and those with outage or accuracy loss in GPS measurements were excluded. Average spatial accuracy of 15 cm was achieved in these outdoor experiments which is greater than previous reports of 50 cm accuracy in cattle sensor systems ,17. HoweBenefitting from the high positioning accuracy, the data analysis performed in the paper was based on the geo-spatial positions of individual animals, which provides more precise information on the interactions between animals than proximity loggers . In addiThe machine learning analyses in this study demonstrated that one animal was central to the group with an influence on other group members. This is similar to other studies of cattle behaviour where certain individuals have social centrality and/or influence on the group’s behaviour ,18. An aThe relationships in the current study were determined over a 25-min period. Previous research from video observations indicated social relationships were moderately to strongly stable across two observation periods six months apart . Rocha eThe results from one study period of a short duration indicated that social interactions could be quantified and animals of different social positions within the group identified. While the devices applied in this study are not yet practically feasible to apply longer-term, or in a commercial scenario, similar to other wireless tracking technologies , there iThe application of the findings from this study and demonstration of the technology and data analysis techniques was limited due to the use of one group for a relatively short period. The period of study (25 min) was restricted by the power requirements as the prototype devices require frequent sampling to ensure high accuracy and this drains the battery and limits the potential for long-term deployment. In addition, the robustness of the prototypes to withstand cattle damaging the casing from rubbing or social interaction are needed for longer-term application in a paddock situation. Another consideration is that no assessment of the impact of the attachment of the girth and devices was conducted and while cattle did not appear to be agitated or affected by the attachment of the devices, there is the possibility that social behaviour may have been impacted. With the development of a more robust weatherproof device with greater battery power, a widespread application could be achieved.In this paper, based on the location information acquired by cooperative wireless localisation, machine learning algorithms were applied to analyse the social structure of a herd of cattle and the social behaviour of the individuals. These algorithms mined the location information related to social attraction and repulsion and were able to quantify the affinity of cattle for one another within a group. Social networks were present in the small herd of cattle, and the localisation devices provided more accurate information than would otherwise be obtained from just video recordings or live observations, such as animals with greater affinity and influence on others. Replication with the devices across more groups and repeatability within groups across different settings would confirm the value of quantifying social interactions with this degree of precision. Further work is needed to apply the devices for extended periods of time, as well as improve the robustness of the devices to be deployed on animals.


Gait and balance impairments are common in neurological diseases, including stroke, and negatively affect patients’ quality of life. Improving balance and gait are among the main goals of rehabilitation. Rehabilitation is mainly performed in clinics, which lack context specificity; therefore, training in the patient’s home environment is preferable. In the last decade, developed rehabilitation technologies such as virtual reality and augmented reality (AR) have enabled gait and balance training outside clinics. Here, we propose a new method for gait rehabilitation in persons who have had a stroke in which mobile AR technology and a sensor-based motion capture system are combined to provide fine-grained feedback on gait performance in real time.The aims of this study were (1) to investigate manipulation of the gait pattern of persons who have had a stroke based on virtual augmentation during overground walking compared to walking without AR performance feedback and (2) to investigate the usability of the AR system.We developed the ARISE  system, in which we combined a development version of HoloLens 2 smart glasses (Microsoft Corporation) with a sensor-based motion capture system. One patient with chronic minor gait impairment poststroke completed clinical gait assessments and an AR parkour course with patient-centered performance gait feedback. The movement kinematics during gait as well as the usability and safety of the system were evaluated.The patient changed his gait pattern during AR parkour compared to the pattern observed during the clinical gait assessments. He recognized the virtual objects and ranked the usability of the ARISE system as excellent. In addition, the patient stated that the system would complement his standard gait therapy. Except for the symptom of exhilaration, no adverse events occurred.This project provided the first evidence of gait adaptation during overground walking based on real-time feedback through visual and auditory augmentation. The system has potential to provide gait and balance rehabilitation outside the clinic. This initial investigation of AR rehabilitation may aid the development and investigation of new gait and balance therapies. Many neurological diseases lead to impairments of gait and balance . ApproxiTherefore, improving gait and balance is one of the main goals of stroke rehabilitation. Repetitive practice is an essential ingredient of established evidence-based interventions  such as Considering the importance of performance feedback for motor skill learning , AR combHere, we propose a new method for gait and balance rehabilitation in patients who have had a stroke that has potential to provide challenging and personalized gait and balance therapy with auditory and visual performance feedback based on gait kinematics in an environment that is adjusted to the patients’ needs. We developed the Augmented Reality for gait Impairments after StrokE (ARISE) system, in which we combined a development version of a head-mounted system for real-time visual and auditory feedback and a commercially available sensor-based motion capture system. Subsequently, we evaluated the usability of the AR feedback prototype in a chronic stroke subject with minor gait and balance impairments. We hypothesized that this system is more capable of modifying gait kinematics than walking without the system.Ethical clearance to execute the experiment with the presented subject was provided by the Cantonal Ethics Commission of the Canton of Zurich, Switzerland (BASEC-Nr. Req-2019-00758). The participant received information about the experiments. Written informed consent in accord with the Declaration of Helsinki was obtained from the participant prior to performing the experiments.The patient was a right-handed man aged 74 years who had a right-hemispheric ischemic stroke in the thalamus, capsula interna, and right temporal lobe 7 years before participating in the experiment. Acutely after stroke, he had mild motor deficits in the left leg . At the time of this experiment, he had minor limitations of sitting and standing balance ; he also had mild limitations in motor function  and strength  of the affected left side. The patient walked independently without a walking aid , with a comfortable walking speed of 1.0 m/s and a step length of 0.56 meters as measured by the 10-Meter Walk Test. The patient had slight risk of falling (Dynamic Gait Index 19/24). He reported numbness in the feet and a feeling of wearing socks when he was not. In addition, a mild cognitive impairment was present .The ARISE system consisted of two essential components: an optical see-through head-mounted display (OST-HMD) and a sensor-based motion capture system (We identified the HoloLens 2 (Microsoft Corporation) as the most suitable OST-HMD, as it provides a wider field of view (43×29 degrees) compared to other devices and thus can display more virtual objects in a real-world environment. In addition, the HoloLens 2 is able to track head movements with an inertial measurement unit and has an intuitive hand-interaction user interface that is enabled through fully articulated hand tracking. The OST-HMD was used to visualize the AR parkour course  A-C. TheIn addition, the OST-HMD provided visual and auditive feedback based on real-time gait kinematic performance. An inertial measurement unit–based motion capture system, the Xsens MVN (Xsens Technologies B.V.), was chosen to track the kinematics of the lower limbs during gait. We strapped seven inertial measurement units on the pelvis and the lower extremity of the patient . The XseThe HoloLens 2 and Xsens systems were integrated by streaming movement performance data to the OST-HMD through a user datagram protocol client. To adjust for translational drift of the motion capture system in the ARISE system, the captured motion data were rigidly attached to the pose of the HoloLens 2. To synchronize the forward directions of both systems, we performed a short calibration phase where the user was required to align his head with his hips and feet. Once the 2 systems were coupled, we were able to provide subtle positive reinforcement of successful knee flexion (>45 degrees) by playing an ambient bird song sound through the HMD.To increase the difficulty of the tasks the subject was asked to complete, simple math calculations were presented visually  of the HoloLens 2 should be improved to increase the usability of the system. The field of view forced the patient to bend forward; this can induce adverse events, such as near-falls or neck pain, when using the system over longer periods of time.Through the coupling of the OST-HMD and the motion capture system, the patient perceived auditive feedback based on the kinematics of the knee (flexion angle). The OST-HMD provided feedback in the form of bird songs and a knowledge of results display. When the patient was given multimodal feedback, he changed his gait pattern. This is in line with previous results ,20. A trThe current AR parkour course has a size of 14×4 meters; therefore, it requires a large open space to operate. In future versions, the ARISE system will be able to generate a parkour course with an adaptable area depending on the local environment. This will provide the advantages of flexibility in choosing the test grounds and a variety of parkour designs. In future research, the impacts of different virtual obstacles or even obstacle themes  on the gait kinematics can be evaluated. Due to the high mobility of all involved components, our system can be used virtually anywhere, both indoors and outdoors and with different surfaces and distractors. The mobility of the system is also ensured through the web interface of the OST-HMD and the motion capture system, which allows remote access. Therefore, a telerehabilitation approach is conceivable.The present ARISE system is a prototype, and it requires technical support for setup and calibration. Therefore, a reduced sensor set and simplified calibration procedure would increase its usability and applicability. Furthermore, movement sensors are limited by orientation drift when they are used over a long period of time . In thisThe ARISE system combines an adjustable and personalized training environment with a feedback and monitoring system for gait and balance rehabilitation. This case study showed that use of the system by patients who have had a stroke and who have mild gait and balance impairments is promising. Compared to existing AR systems, the system utilizes the real physical environment to place virtual obstacles that are immediately and intuitively treated as real-world objects. This system can be used to provide feedback on a variety of gait parameters and to implement personalized  gait training environments.


Progress control is a key technology for successfully carrying out a project by predicting possible problems, particularly production delays, and establishing measures to avoid them (decision-making). However, shipyard progress management is still dependent on the empirical judgment of the manager, and this has led to delays in delivery, which raises ship production costs. Therefore, this paper proposes a methodology for shipyard ship block assembly plants that enables objective process progress measurement based on real-time work performance data, rather than the empirical judgment of a site manager. In particular, an IoT-based physical progress measurement method that can automatically measure work performance without human intervention is presented for the mounting and welding activities of ship block assembly work. Both an augmented reality (AR) marker-based image analysis system and a welding machine time-series data-based machine learning model are presented for measuring the performances of the mounting and welding activities. In addition, the physical progress measurement method proposed in this study was applied to the ship block assembly plant of shipyard H to verify its validity. To improve productivity and enhance cost competitiveness, the shipbuilding industry aims to establish smart shipyards that can manage performance through the real-time collection of production information. Progress control is a key technology for successfully carrying out a project by predicting possible problems, such as production delays, and establishing measures to avoid them (decision-making) ,2,3.Recently, the shipbuilding industry has become more complex in terms of production site management due to the mixed production practices caused by the diversification of ship types and shapes. As a result, production schedules and executions often fail to match, resulting in too much time and money being needed to identify process status, and complicating the immediate actions taken by managers in response to changes in site conditions. Therefore, a rapid and accurate process performance management system is required to allow appropriate actions in response to changes in site conditions.However, shipyard performance management is still based on the task manager’s empirical (intuitive) know-how ,5. When The digital twin/cyber physical system (CPS), Internet of Things (IoT), big data, cloud and robots, which have recently emerged as the wave of the fourth industrial revolution (the Industry 4.0), can present new solutions as alternatives to the various problems and existing empirical performance measurement methods at shipyards ,7,8. In The digital twin is a technology that creates twins  of the physical objects on a computer and predicts what may happen in the real world through simulation . The virTherefore, in order to implement digital twin-based smart shipyards, automated performance measurement methods of physical systems are first required. Studies have recently been conducted into collecting and analyzing data at shipyard production sites in real time and then using that data for production management and performance management.Chang et al.  demonstrPaula Faga-Lamas et al.  proposedLee  describeKim, M. et al.  proposedBlanco-Novoa, O. et al.  describeIn terms of project management, performance measurement research has also focused on the construction industry, which is similar to the shipbuilding industry. Golparvar-Fard, M. et al.  developeAs such, recent studies have been conducted to improve both the convenience and performance of data collection and streamline production management by converging ICT (information and communication technologies) into existing shipyard production processes. However, there is still a lack of research into how real-time performance information can be collected at shipyards without the intervention of workers or objective progress measurement based on production data. Therefore, in this study, an IoT-based performance measurement method is presented that can automatically collect performance data without worker intervention through analyzing the work procedures and facilities used in the ship block assembly process. In addition, a real-time workload-based progress measurement method is proposed that allows objective performance measurement using actual work data collected in real time. Finally, the proposed method was applied to the ship block assembly plant in shipyard H to verify its validity.For this study on automatic performance data collection in real time in the assembly process of shipyard ship blocks, the curved block assembly plant was selected as the subject of analysis, and both the work procedure and process facilities were analyzed. Since generally the curved block assembly plant acts as an overall bottleneck in the ship assembly process, and it has a significant impact on the forward and rear processes, and this plant requires very precise performance control. The following describes the general assembly process at the curved block assembly workshop.PIN JIG setting: Before placing the ship’s outer plates on the jig, the height of the pin jig is adjusted according to the curvature of the ship’s outer plate.Ship’s outer plate arrangement and joining: The ship’s outer plates are arranged on a controlled pin jig and the outer plates are welded. The outer plates are moved by overhead cranes and placed in the workshop. Welding machines are used for joining operations on the outer plates. Mounting: Various materials and subassembly blocks are put on the ship’s outer plates to produce the final shape of the ship block. Tack welding/welding (main welding): Work pieces, such as materials and subassemblies, placed on the ship’s outer plates are fixed in position by tack welding and then the work pieces are bonded by welding. Generally, tack welding is done before the main welding. Grinding: Weld grinding is cleanly removing the excess weld metal. The duration of this grinding activity depends on the quality of welding; good quality welding reduces grinding time.Inspection/transfer: Once the ship block has finished the grinding process, it will be transported to an outdoor waiting area after an inspection. An overhead crane is used to transfer the ship block out of the workshop.The ship block assembly process described above consists mostly of mounting and welding ship block work pieces. Welding activity management needs to consider two distinct activities, tack welding and welding. In general, there are three main types of process progress measurement methods: the estimated percent complete method (EPCM), the earned value method (EVM), and the physical progress measurement method (PPMM) ,2,3.First, the EPCM is a method by which a control manager of a process or activity identifies the progress of work and assigns a progress rate based on his/her subjective judgment. Progress measurement standards are very simple, advantageous for simple repetitive construction and can save time and man-hours. However, they lack objectivity and are likely to be inconsistent with the relationship between the progress of a process and the actual workload.Second, the EVM is a method of calculating progress rates by dividing the unit scope into measurable scales and marking achievement progress by the stage of the work. The more detailed the level of segmentation is, the more objective the measure of progress will be.Third, the PPMM is a method of measuring actual work completed as a percentage of the total planned work volume, allowing meaningful progress rates. Of the three methods of measuring progress, this requires the highest level of objectivity regarding work volume. In practice, measuring the actual workload takes many man-hours and much time.The generally used means of progress management of a shipyard—the subject of this paper—is EPCM based on the empirical (intuitive) judgment of the field manager. This is because the assembly work of a shipyard’s ship block is difficult to break down into measurable levels of work and the manpower and time needed to measure the actual workload are lacking.Therefore, achieving the objective progress measurement and reliable decision-making required by the shipyard requires not only a real-time work performance collection system that can reduce the time and manpower required for collecting work information but also a new method for objective progress measurement based on actual workload.The ship block assembly process consists of the mounting, tack welding, welding, grinding and inspection activities described in j , the mounting, the tack welding, the welding, the grinding and the inspection. Here, the working ratio of activity i in the assembly process of the ship block is different. The completion of activity i is seen as a percentage of work done by activity i in the entire ship block assembly process. Thus, the work-weight constant i in the total workload of ship block j. i of ship block j. The total progress of the ship block is shown in Equation TPj=∑i PAijMethods for measuring the performance of the mounting and welding activities are subject to physical progress measurement in this study are described in As described in Recently, imaging systems have been used as work performance measurement methods in various industries in order to reduce time and manpower required for performance data collection ,27,28. TThus, in this study, an imaging analysis method using a marker is proposed to facilitate analysis of the process progress of ship block mounting activities. The method of image analysis using a marker proposed in this study consists of four steps, as shown in Augmented reality (AR) markers are visual signals which trigger the display of the virtual information . GeneralUsing the camera installed on-site, ship block images were collected and the AR markers in images were analyzed to recognize the components mounted on the ship block. AR markers were printed on steel plates in the preprocessing process of the shipyard.The images of the ship block were taken sequentially in an S pattern from the lower left corner of the workshop based on preset coordinate values. Ship block images collected at the site are transmitted to the server using a wired/wireless network. More than 200 images are created for a ship block. However, the quantity of images collected depends on the size of the ship block and workshop.To simplify image analysis, the collected AR marker images need to be converted to images consisting of only black and white pixels from those consisting of colored or gray pixels. This pre-processing of images is called “image binarization”. In this study, the adaptive thresholding method was used for image binarization ,36. AdapThe AR marker images collected in this study had unclear marker boundaries and contained much noise due to the low light environment of the site. Thus, the images had to undergo a smoothing process before binarization. This study used the Gaussian blur method and the bilateral filter method for image smoothing ,38,39,40The analyzed AR marker is mapped to the unique ID of the ship block material. Therefore, comparing the assembly diagram and AR markers of the ship block shows the current process progress of the ship block being worked on. Here, since the unique ID of the AR marker is the material ID on the assembly diagram of the ship block, if any of the AR markers are detected, it is assumed that the entire component (one intermediate assembly on the assembly diagram) is mounted. One component of a ship block is made of several materials.j (j (j is as shown in Equation (The progress of the mounting activity (Equation .(4)PAijThis chapter proposes a method for automatically measuring the performance of welding work using sensor data, particularly that of current, voltage and weld-wire speed, generated by welders. As in the ship block assembly process tack welding and welding account for more than 60% of the total workload and are time-consuming, it is very important to objectively manage the performances of welding activities. Tack welding and welding are done by different workers. Available budgets are also managed separately. Thus, this study proposes a machine learning-based welding type classification model that can automatically classify tack welding and welding work through supervised learning from sensor data. The proposed method consists of the following procedures shown in In this study, to collect the working data of the welder, an electric current sensor and voltage sensor and a welding wire feed speed sensor were attached to welding machines and a unique ID was given to the weld feeder. After collecting the time series welding data from each sensor, the data are sent to an IoT gateway. The IoT gateway combines the received welding machine sensor data and time data and sends them to an analysis server. The data are transmitted every second. This section presents the welding type classification model and evaluates its performance at automatically distinguishing sensor data collected from welding machines doing tag welding or normal welding.Using real-time welding time series data, the proposed method of measuring sensor-data-based tack welding/welding activity performance predicts welding types and calculates the performance of the predicted welding work. To predict the welding type from the welding time-series data  collected from the welding machines, first the respective features of each type of welding operation need to be identified in the time series data. In ship block assembly work, the types of welding include tag welding, welding  and repair welding. Automatic welding is welding using an automatic welding carriage machine, and manual welding is welding done manually by an operator using a welding torch.In this study, the electric current (ampere), voltage and weld-wire feed speed of each welding operation were visualized, as shown in Looking closely at the time series graph for each type of work, automatic welding has the characteristics of long continuous working hours and small variations in sensor data. Tack welding and repair welding have short continuous working hours and large variations in sensor data. Manual welding has a complex mix of characteristics found for automatic welding, tack welding and repair welding, has some continuous working time and has a large variation in sensor data.In this study, session window techniques were used to transform the time series data collected from welding machines into datasets suitable for supervised learning. Windowing refers to the way in which time series data are processed in real time and windowing techniques include tumbling (or fixed) windows, hopping (or sliding) windows and session windows ,42,43. Fi is the work number.To implement the welding type classification model through supervised learning, sensor data from welding machines and welding work type data are required, where the sensor time series data are used as the independent variables for the model, and the welding operation type data are used as the dependent variables. In this study, the time series graph shows that features such as sensor data value variation over time or continuous working time are important factors for distinguishing between the different types of welding work. Thus, the statistical information of the sensor data is used as an independent variable for the welding type classification model. First, as shown in Equation , the aveIn order to calculate the variation in sensor data during continuous working hours, the current variance Equation .(6)vai,The variable Equation .(7)csi=In addition, the current standard deviation, the voltage standard deviation, the weld-wire feed speed standard deviation and weld-wire length were extracted as feature variables. Next, to define the feature variables to be used for the weld type classification models, the correlation between the feature variables and the welding types were identified. i for learning the weld type classification model. The formulas of the feature vector In quations –10). . Table 2In F1 score (F-score or F-measurement) is a measure of the accuracy of the test as a harmonized average of precision and recall [F1-score is also used to evaluate the performance of a model when the data label is unbalanced.Accuracy above is the evaluation indicator that can most intuitively represent the performance of the model. It indicates the probability that the results of the classification model will match the actual value. The d recall ,46. The In this subsection, the weld type classifiers are tested and evaluated using actual welding data from shipyard H block assembly plant. The experimental procedure consists of two steps. First, (1) use the training datasets to learn the welding type classification model. Then, (2) enter the test datasets into the welding type classification model to evaluate the performance of the classification models. About six months of time series data collected from welding machines at the shipyard H (block assembly plant) were used as data for experimentation and verification. Four months of the data were used as a training dataset and two months as a test dataset. In this study, the performance of the model was compared and evaluated using the ensemble method, which has been used most often in recent machine learning-based classification models as a classifier of the welding type classification model ,48. The Next, to verify the performances of the five case models selected in this study, a set of test data was entered into the model and the results were written in a confusion matrix. Comparing the actual data by welding type with the predicted results by classification models reveals that automatic welding and manual welding are relatively well classified, but that some data are misclassified for cases of tack welding and repair welding. However, each model has different performance advantages for predicting welding type, so it can be supplemented.Thus, in this study, stacking model (STK) was also implemented to achieve higher recognition performance and robustness than ultimately utilizing the results of the individual models . The staThe performance of the classifier calculated using the confusion matrix is shown in j, it is necessary to calculate the workload of the tack-welding/welding of each welder l working in ship block j. Therefore, the process progress of the tack welding/welding activity (l performed in ship block j (j (To measure the workload of tack-welding/welding in ship block Equation .(15)PAiIn order to validate the method proposed by this study, the schedule variance of the workplace with and without the proposed method (system) was compared and evaluated over the same period. Four work teams were selected and compared at each workshop. Schedule variance (SV) is the difference between the planned value (PV) and the earned value (EV) . SV < 0 From these results it can be said that the work teams that actively performed process control through the methods presented in this study have excellent schedule compliance.In this paper, a methodology that enables objective process progress measurement based on data, rather than the empirical judgment of a site manager, was proposed for shipyard ship block assembly plants. To this end, a method was defined for the measurement of work progress by activity that allows for the measurement of overall ship block progress performance. In particular, IoT-based performance collection methods were proposed for mounting and welding activities, which account for most of the work in ship block assembly, and those methods were found capable of automatically measuring actual work volume without human intervention. An AR marker-based image analysis system was proposed as an automatic performance collection method for mounting activity. As part of the method of collecting welding performance activity, a machine learning-based performance measurement method that can analyze performance using the sensor data of a welding machine was presented. In addition, the proposed method was applied to the ship block assembly plant at H Shipyard to verify its validity.The method presented in this paper means that the collection and processing of on-site shipyard information can be automated, thereby reducing the time and manpower required for the collection of production information and enabling objective and precise performance management. Therefore, the workload-based progress measurement method presented in this paper makes the following contributions to ship block process management. First, it is expected to improve the capability of process management by using PPMM instead of the EPCM by site managers currently used in the shipyard ship block assembly process. Second, by presenting a way to collect actual working data in a shipyard environment directly by utilizing IoT technology, the manpower required for performance collection can be reduced, and the accuracy of the performance information collected can be improved. Third, it is expected that various analysis services based on real-time production data will become available through vertical integration through the physical world and the cyber world, accelerating the implementation of smart shipyards. In addition to this study, more process innovation research applying IoT, big data and AI technologies to shipbuilding will be needed to strengthen the competitiveness of the industry and improve productivity.In future research, we would like to conduct a study of dynamic scheduling operating on the basis of the work performance automatically collected in this study. Scheduling is a very important area of research for improving production and strengthening shipyard competitiveness. Accordingly, many studies related to scheduling have been carried out by various institutions. Nevertheless, those studies have been mainly conducted on static scheduling because they were based on the assumption that performance was adequately reflected or entered at the time the manager made the plan. However, since this study shows how performance data at the production site can be automatically collected in real time, it is possible to study the dynamic scheduling of changes in real-time production sites. Moreover, it is expected that integration with the results presented in this study will realize the completion of CPS for shipyard assembly block plants.


In this study, a multistage segmentation technique is proposed that identifies cancerous cells in prostate tissue samples. The benign areas of the tissue are distinguished from the cancerous regions using the texture of glands. The texture is modeled based on wavelet packet features along with sample entropy values. In a multistage segmentation process, the mean-shift algorithm is applied on the pre-processed images to perform a coarse segmentation of the tissue. Wavelet packets are employed in the second stage to obtain fine details of the structured shape of glands. Finally, the texture of the gland is modeled by the sample entropy values, which identifies epithelial regions from stroma patches. Although there are three stages of the proposed algorithm, the computation is fast as wavelet packet features and sample entropy values perform robust modeling for the required regions of interest. A comparative analysis with other state-of-the-art texture segmentation techniques is presented and dice ratios are computed for the comparison. It has been observed that our algorithm not only outperforms other techniques, but, by introducing sample entropy features, identification of cancerous regions of tissues is achieved with 90% classification accuracy, which shows the robustness of the proposed algorithm. There are around 700,000 deaths from prostate cancer per year worldwide, and it is one of the top five cancers that cause fatality ,2. SimilIn digital pathology, digital cameras with high resolutions are used to capture the images of pathology slides. A huge amount of data is generated when digitization of glass slides is performed. There are a number of advantages for digitization that include expediate transmission, collective reporting, and forum discussions. A few challenges for digitization include lighting and staining artifacts, processing of a large amount of data and its storage, along with the absence of standards . HoweverIn this study, we have developed an automated algorithm using texture based on sample entropy to identify cancerous tissues of prostate cancer. Texture is a feature that is determined by the spatial arrangement of the intensity levels of pixels in a closed neighborhood. It is a pattern that repeats locally for variations of the intensity of an image . In a fiIn our proposed algorithm, a multistage segmentation process for glandular structures of epithelial tissue, choosing prostate tissue as an example, is proposed. There are three stages that work in sequence to model the texture of tissue glands. After doing a few pre-processing steps, such as affine transformation and morphological operations including closing and top hat transforms, a coarse segmentation of the input images is performed using mean-shift segmentation. In the next stage, wavelet packets are used that extract glandular regions of the tissue. The texture of the extracted regions is modeled using sample entropy values and, applying a non-linear threshold, cancerous tissues are differentiated from benign tissues. The block diagram of the proposed system is presented in The organization of the paper is presented as follows. Related work and background for the proposed algorithm is introduced in the next section. A variety of algorithms have been proposed in the literature for the analysis of colon and prostate histology images , butIn boundary segmentation methods, Naik et al.  have useA few regions-based approaches for glandular segmentation, such as object graphs  and iterNguyen et al.  proposedIn the work of , a stack(1)As a first step, segmentation of the tissue is achieved by applying the color mean-shift (MS) algorithm  on the i(2)The above segmentation result can be used to get a rough idea of pixels forming nuclei, both stromal and epithelial. In the second step, textural features for pixels of the image are used to model the glands. The wavelet packet features  and AdaB(3)Finally, the texture of the image is modeled using sample entropy analysis. Using empirical methods, a threshold is obtained that distinguishes cancerous regions form the benign tissue samples.Modeling of the glandular structures in the proposed algorithm, as shown in Our hypothesis is that glandular structures are formed by epithelial nuclei at the boundary of glands and the lumen is inside them. We employed the color mean-shift (MS) algorithm  for a co2n, where n is the level of decomposition. In the wavelet packet analysis, the image is examined at different levels. The over-complete wavelet packet decomposition provides a feature vector that is computed from the energy and entropy features of the texture that is obtained by wavelet packets and sample entropy analysis. In wavelet packet decomposition, the approximate and detail images at each level are decomposed into four levels and a quad tree image filter is achieved. In conventional wavelet packets transform, the down-sampled sub images are one-quarter of the size of the original image. However, in over-complete filter decomposition, there is no down-sampling and the sub-bands are of the same size as the original. Thus, a robust feature vector is obtained. A high number of features can be obtained with an increase in the decomposition level, as the number of sub-bands increases exponentially. In two-level decomposition, eight feature sub images are retained only and these images contain maximum variance in each level.In wavelet packets, both low and high frequency components are decomposed in each sub-band. An analogy of binary trees is presented for wavelet packets where every low or high sub-band represents a node with two children. As the binary tree may have either 0 or 2 children, the basis of wavelet packets is 2In sample entropy, the spatial distributions of pixels are correlated to compute the entropy distributions . The entmP(d) and mP+1(d); by determining their logarithmic ratio, the sample entropy can be computed. An image I  with in rows and jn columns is considered that contains kx  set of pixels varying from j to j + m − 1 column lines and i to i − m + 1 row. The window size is kx  and it is centered at i and j. There are only (in − k) × (jn − k) patterns as the last m lines cannot be used for square patterns as the total number of pixels is equal to N, where N = in × jn. To determine the mathematical formulation of the sample entropy, the following derivations are used.pN is the total number of m length different squared patterns in an image, whereas mP and mP+1 are m and m + 1 length pattern in the image, respectively. The sample entropy is the logarithmic ratio for these patterns. The entropy and energy are computed for each segmented sub image with the square local area obtained by sample entropy window size. Energy is the content of information in an image. It is defined on the normalized histogram of an image and it shows how gray levels are spread in the image. The energy of an image is inversely proportional to Shannon entropy and is closely related to randomness. A high energy image has a low number of gray levels. For each squared area, the feature vector represents values for the central pixel of the sub image.Sample entropy is a measure of irregularity and periodic or regular patterns show small values of sample entropy. Let us consider two patterns We use texture as a cue for assigning lumen-ness labels to all pixels in the image. The wavelet packet transform is used to compute pixel-based features. In the wavelet transform, the approximation filter is not divided and only the last level is split for the high levels of approximation and detailed images. The idea behind the wavelet packet transform is to further decompose both approximation as well as detail sub-bands at each but the last level. The advantage is that useful information in the high signal wavelengths is computed for the richer analysis of textural features. The wavelet packet features perform fine segmentation of the tissue. AdaBoost is added in the model that enhances the performance of the classifier. In AdaBoost modeling, weak classifiers are converted into a strong classifier.Daubechies 4 filters of wavelet packets are used to extract a 64-dimensional feature vector. Pixel labeling algorithm using wavelet packet features is trained on 50 random samples in the training data. There were 25 prostate cancer (suspected) patients and 2 samples were collected from each patient. We trained and tested the proposed technique on a dataset of 50 Hematoxylin and Eosin (H&E) stained prostate histopathology images. The dataset is collected from National Cancer Institute (NCI), where regions of interest (ROIs) from whole slide images are extracted at 40X magnification . The resAn average patch size of 50 × 50 pixels was selected such that half of them were from lumen areas and the other half were from the stromal areas for each of the training images. The wavelet packet transform for the third level generates features for the energy and entropy of each patch and a 64-dimensional feature vector is produced. The Adaboost classifier is used to enhance the accuracy of SVM. The Adaboost technique enhances the performance of a weak classifier, which has accuracy of slightly more than guessing, into a strong classifier. In each iteration of the Adaboost algorithm, the weight of the misclassified observation is increased, otherwise the weight is reduced. In a few iterations, the training error is reduced exponentially, and the algorithm achieves rapid convergence. The trained classifier is then used to assign pixel labels to each image pixel in our dataset based on their 64-dimensional wavelet packet features.To test the quality of the segmentation results by our algorithm, ground truth glandular regions are computed for each gland marked with its boundary. In TP represents total number of positive samples that are also positive in the ground truth, whereas FP is the false positive cases, which are false in the ground truth, but predicted to be true by the algorithm. FN represent the false negatives that the algorithm predicts to be negative, while in the ground truth, they are positive. To evaluate the performance of our algorithm, the Sorensen–Dice index or Dice Similarity Coefficient (DSC) is used as a quality measure with its value ranging between 0 and 1 . It is aTo evaluate the experimental results, we have computed quality metrics such as accuracy, precision, and recall. They are defined as follows:The parameters in the confusion matrix are explained as follows.TP): The algorithm detects it as cancerous and the ground truth also labels it as cancerous.True-positive (FP): The algorithm detects it as cancerous, while the ground truth labels it as normal.False-positive (TN): The algorithm detects it as normal, whereas ground truth also labels it as normal.True-negative (FN): The algorithm detects it normal, whereas ground truth labels it as cancerous.False-negative  algorithm is used to perform the coarse segmentation to split the tissue constituents in four parts. In the second stage, wavelet filters are used to perform fine segmentation of the tissue glands. For classification, sample entropy features are extracted from the segmented regions along with the energy parameters. It has been observed that sample entropy achieves more than 90% classification accuracy. An added advantage of performing multistage segmentation using sample entropy values is that one could easily separate epithelial nuclei from the stroma nuclei in standard H&E stained images without using any additional immunohistochemical (IHC) markers. Furthermore, morphological signatures of glandular structures modeled using our algorithm can also be fused with disease signatures from other modalities such as mass spectrometry (MS), IHC, flow cytometry (FC), and/or cytogenetic (CG) testing for integrated diagnostic/prognostic purposes ,42. Futu


In this paper, we synergistically combine electrohydrodynamic (EHD) printing and replica molding for the fabrication of microlenses. Glycerol solution microdroplets was sprayed onto the ITO glass to form liquid mold by an EHD printing process. The liquid mold is used as a master to fabricate a polydimethylsiloxane (PDMS) mold. Finally, the desired micro-optical device can be fabricated on any substrate using a PDMS soft lithography mold. We demonstrate our strategy by generating microlenses of photocurable polymers and by characterizing their optical properties. It is a new method to rapidly and cost-effectively fabricate molds with small diameters by exploiting the advantages of EHD printing, while maintaining the parallel nature of soft-lithography. Microlens and microlens array (MLA) with diameters of a few to hundreds micrometers have wide applications in various fields, including artificial compound eyes , light-eInkjet printing and laser-induced transfer provide methods of fabricating microlens molds or directly fabricating microlens arrays ,10. The The soft lithography method is a method of performing microstructure replication or patterning polymer droplets using a PDMS elastomer mold ,13,14. BEHD printing is one of the most promising micro-nano additive manufacturing technologies with its unique advantages of low cost and high resolution ,20,21. IIn this work, we propose a new method for fabricating microlens arrays that combines the advantages of soft lithography and EHD printing. We use a drop-on-demand EHD printing technique to print glycerin liquid on the surface of ITO glass. The glycerol liquid droplets printed on the ITO glass substrate are hemispherical due to surface tension, thus forming microlens liquid mold. The PDMS soft lithography mold is then quickly fabricated using a liquid mold. Finally, the desired micro-optical device can be fabricated on any substrate using a PDMS soft lithography mold. Therefore, our methods allows us to rapidly and cost-effectively fabricate molds with small diameters by exploiting the advantages of EHD printing, while maintaining the parallel nature of soft-lithography and its capacity to generate microlenses of different materials. We also discuss the effects of the main process parameters on the stability of the printing process and the diameter of the droplets.The 80 vol% glycerol solution  was selected to be the ink materials for EHD printing the micro-droplet array. Sylgard 184 elastomer base and curing agent for PDMS were purchased from Dow Corning . The weight ratio of silicone elastomer to curing agent is 10 to 1. The UV-curable prepolymer  is used as the material for replicating microlens.The schematic diagram of the experimental EHD printing system for printing liquid microdroplets is shown in Before the experiment, the ITO glass substrate was ultrasonically cleaned in acetone for 20 min and then dried with air flow. Glycerol solution was first sprayed onto the ITO glass by an EHD printing process. The microdroplets form liquid mold on the surface of ITO glass. The spacing, diameter, and volume of the microdroplets liquid mold can be controlled by processing parameters. The PDMS mold for the microlens fabrication was obtained by casting PDMS mixture (the weight ratio of silicone elastomer to curing agent is 10 to 1) into the ITO glass that contained the EHD printed glycerol micro droplets liquid mold. The cast PDMS was then placed in a vacuum oven at 70 °C for 3 h to fully crosslink it to obtain a PDMS microlens mold. Then take it out, rinse off the surface glycerin, and then dry in an oven for 1 h. The UV crosslinked resin was cast on the surface of any substrate, and then covered with a PDMS mold for solution casting, and irradiated with a 36 W UV lamp  for 3 min. Finally, the PDMS mold was peeled off to obtain a microlens array formed by curing the UV resin . The imaging performance of the microlens was tested by a projection experiment, and an image formed by the microlens was observed using an optical microscope .Understanding the process of jet formation and determining the key parameters that affect the process are critical to high resolution, uniform, and reproducible printing. In order to obtain a good morphology and meet the requirements of the microlens liquid mold, we explored the drop on demand EHD printing process of printing glycerol solution.We first explored the parameters that affect the jet formation. The pulse jet formation process is strongly dependent on the distance between nozzle and collector, glycerin solution concentration, pulse voltage, and bias voltage.The concentration will significantly affect the conductivity and viscosity of the solution, which will have a large impact on the stability of the printing process. When the concentration of the aqueous glycerin solution is high, the viscosity is large and the electrical conductivity is weak, and it is difficult to form a stable pulse jet. When the concentration is too low, the conductivity is enhanced and the viscosity is lowered, which causes a large number of tiny satellite droplets around the main droplet. The printing process is converted into electrospray , so thatElectric field is a key factor affecting the formation of a stable pulse jet. It depends on the distance from the needle to the collecting plate and the type and magnitude of the voltage applied to the needle. When the voltage is fixed, if the distance from the needle to the collecting plate is too large, the electric field strength is insufficient to form a jet. However, if the distance is too small (less than 300 μm), corona discharge would occur and prevent the generation of jets. Therefore, setting a suitable distance is critical to the stability of droplet ejection. In this work, the distance was set to a range of 800 μm to 1200 μm to overcome the corona discharge.When a conventional pulse voltage is applied to the needle for EHD ejection, the lower voltage is insufficient to form a stable pulse jet due to the higher viscosity of the aqueous glycerin solution. However, increasing the pulse voltage or lowering the viscosity of the solution will lead to other undesirable consequences such as corona discharge or a large number of satellite drops. Therefore, in order to achieve a stable on-demand droplet printing of an aqueous glycerin solution, we applied a bias voltage of −1 kV to the ITO glass collecting plate, as shown in We also explored the parameters that affect the diameter of the microdroplets dot pattern. It is mainly affected by pulse voltage frequency, pulse duty cycle, flow velocity of solution, and pulse voltage amplitude.pD can be calculated byplateV is the collecting plate velocity, f is the frequency of the pulse voltage.The frequency of the droplet ejection is equal to the frequency of the pulse voltage. The spacing of the two droplets ejected onto the substrate can be precisely adjusted by controlling the pulse frequency and the moving speed of the collecting plate. The spacing of the two dot dotD is the spherical cap (dot pattern) diameter, dropD is the droplet diameter, and θ is the contact angle.Stinger and Derby  studied dropV is the total droplet volume ejected from the Taylor cone during each pulse period. Q is the flow rate, T is pulse width time, d is the duty ratio, and f is the pulse frequency.The drop diameter can be calculated as followsTherefore, deposited pattern diameter can be expressed as (6)Ddot=As the pulse frequency increases, the time during which each injection cycle voltage acts on the Taylor cone becomes shorter, the accumulated charge and the volume of the solution decrease, and thus the diameter of the droplet decreases. Therefore the frequency of the pulse voltage will significantly affect the dot pattern diameter, as shown in d1/3.We kept the pulse period fixed and increased the voltage duty cycle and found that the droplet diameter increased. When the duty time increased, the volume of the solution and the accumulated charges on the taylor cone increased, so that the repulsive force became stronger, which resulted in a larger diameter microdroplets. Therefore, the duty cycle of the pulse voltage per injection is a key factor affecting the droplet diameter and dot pattern diameter, as shown in dot, QD isWe further studied the effects of pulse voltage on droplet diameter. Finally, we combined the PDMS soft lithography process to prepare microlenses based on liquid molds. The PDMS mixture (the weight ratio of silicone elastomer to curing agent is 10 to 1) was slowly poured onto the liquid microlens mold  was used for micro lenses array projection experiments. The mask we inserted with the letter “F” is printed on a black paper between the bright light source and the micro lenses, as shown in the In summary, we propose a simple, economical method to rapidly fabricate microlens arrays based on liquid mold soft lithography processes. Compared to other inkjet printing technologies, our DOD EHD process can print smaller droplets at a higher speed. In addition, once the PDMS concave microlens mold is fabricated, it can be reused for microlens replication without having to go through the entire process again. We also studied the process parameters that affect the stability of the printing process and the diameter of the printed micro droplets. By applying a biased DC voltage on the collector plate, it helps to form a stable pulse jet, avoiding satellite drops or corona discharge. This work was expected to promote the application of EHD printing technologies in the fields of manufacturing of micro system.