We report the development of a new type of space lidar specifically designed for missions to small planetary bodies for both topographic mapping and support of sample collection or landing. The instrument is designed to have a wide dynamic range with several operation modes for different mission phases. The laser transmitter consists of a fiber laser that is intensity modulated with a return-to-zero pseudo-noise (RZPN) code. The receiver detects the coded pulse-train by correlating the detected signal with the RZPN kernel. Unlike regular pseudo noise (PN) lidars, the RZPN kernel is set to zero outside laser firing windows, which removes most of the background noise over the receiver integration time. This technique enables the use of low peak-power but high pulse-rate lasers, such as fiber lasers, for long-distance ranging without aliasing. The laser power and the internal gain of the detector can both be adjusted to give a wide measurement dynamic range. The laser modulation code pattern can also be reconfigured in orbit to optimize measurements to different measurement environments. The receiver uses a multi-pixel linear mode photon-counting HgCdTe avalanche photodiode (APD) array with near quantum limited sensitivity at near to mid infrared wavelengths where many fiber lasers and diode lasers operate. The instrument is modular and versatile and can be built mostly with components developed by the optical communication industry. Lidar (light detection and ranging) instruments have been used to measure surface elevation, shape, tidal deformation, spin axis, and reflectance in planetary missions ,2,3,4. LSmall planetary bodies are remnants of planetary formation. They preserve constituents that are indicative of conditions in the early Solar System in mostly pristine settings. Numerous astronomical surveys of asteroids ,11,12 haLidars for missions to small planetary bodies have different requirements than those for major planets because of the lower orbital altitudes and slower ground track speeds. They do not need to have high-power lasers and large receiver aperture sizes. The major technical challenges in the lidar design are its flexibility and dynamic range because of the irregular shape of the body and the wide range of orbit altitudes. The lidar should provide a coarse measurement of the shape and rotation axis of the body from a high altitude during the reconnaissance phase of the mission prior to high precision mapping. It is also highly desirable that the same lidar can provide real time spacecraft altitude and velocity data to the spacecraft GN&C system during descent and touchdown. A multi-beam or scanning lidar is preferred to improve the spatial coverage and provide surface slope and roughness measurements.There have been several lidars flown to asteroids over the past 25 years ,7,8,9. TSALi is specially designed for missions to small planetary bodies. It features three new technologies. First, a RZPN code is used to modulate the laser transmitter. The receiver detects the laser pulse pattern instead of individual laser pulses by correlating the received signal with the RZPN kernel . This teSALi is designed to operate in three modes: a coarse survey mode from a long distance, a precision mapping mode from a mid-altitude orbit, and a real-time GN&C sensor mode for touchdown and sample collection. The primary science measurements are conducted in the mapping mode. The multi-beam measurements from different orbits create large number of cross-over measurement points which can be used to refine the spacecraft orbit position and laser pointing angles  and consThis paper is organized as follows. PN codes have long been used for radio frequency (RF) ranging systems, such as in the Global Positioning System (GPS). PN code lidars have previously been used for atmospheric profiling and ranging measurements ,32,33,34waveform ,37. The Use of PN code modulation enables lidar to use lasers with lower peak powers for long distance ranging. The receiver sensitivity increases with the received integration time, which allows a tradeoff between the maximum range and measurement rate. A major shortcoming of a conventional PN code lidar is its susceptibility to background illumination and detector dark noise. The receiver for a conventional PN code lidar integrates the background and detector dark noise with the signal for the entire receiver integration time, which is often the dominant noise source.We have developed a variant of the PN code and kernel, which retains all the advantages of a regular PN lidar but is much less susceptible to background noise . In thisThere are three major advantages to the RZPN lidar technique. First, it nulls out all background noise in the return-to-zero time period when calculating the correlation function. Second, it uses shorter laser pulse width which helps to improve the ranging precision. Third, it provides space in the remaining bit intervals for additional measurement channels without direct interference as long as they use different RZPN codes. This laser modulation technique provides a new degree of freedom in PN code lidar design, namely, the laser pulse width. This approach is especially suited for fiber lasers where the pulse width and pulse rate can be varied over a wide range under the same average laser output power and the electrical power. The average output power can also be adjusted by changing the pump laser power. Fiber lasers have a much higher electrical-to-optical conversion efficiency than Nd:YAG lasers. They are small and modular, and widely used in the telecommunication and other industries. They have advanced rapidly in recent years.Assuming a rectangular pulse shape, a transmitted laser signal in a RZPN code lidar over one code length period can be expressed asThe received signal is the convolution of the transmitted signal and the target impulse response function, The normalized correlation function of the received signal and the kernel is given bySince the convolution and correlation are linear operations, their order can be exchanged and Equation (3) can be rewritten asTherefore, the cross-correlation function of a RZPN lidar is equal to the convolution of the target impulse response with the normalized cross-correlation function of the transmitted RZPN code with its kernel given in Equation (8). For a rectangular laser pulse shape, the peak of this cross-correlation function is a triangle with the base equal to two times the laser pulse width. It is a periodic function with the period equal to the PN code length. The impulse response of the target can be much wider than the laser pulses. It broadens the shape of the peak of the correlation function, but it does not cause aliasing.In practice, the received signal is digitized and so is the kernel. Assuming the signal is digitized at interval As mentioned earlier, the signal is sampled at a fraction of the laser pulse width, i.e., For relatively weak signals, the signal or the correlation results need to be averaged to achieve a sufficient signal-to-noise ratio (SNR). The order of averaging and correlation can be exchanged without affecting the result since they both are linear operations. In practice the signal is usually averaged first, since signal averaging can be carried out relatively easily and it reduces the data volume to be carried forward.Finally, a peak search is performed on the outcome of the cross-correlation function. The centroid of the peak gives the target distance, the width of the peak gives an indication of the laser pulse spreading by the target slope and roughness, and the integral of the peak gives the received laser energy, which can be used to infer the surface reflectance.The RZPN lidar technique enables the use of low peak power lasers for long distance ranging. The efficiency of a PN or RZPN lidar in terms of the average transmitted laser power required to achieve a given receiver SNR is less than a single pulse detection lidar , but it PN and RZPN lidars are generally far less susceptible to laser speckle noise than conventional lidars. Laser speckle can be a significant noise source for conventional lidar using small laser footprint sizes and low pulse rates . Laser sRZPN code modulation has an additional benefit of allowing time division multiplexing of other measurement channels during the return-to-zero time periods. For example, one can insert another measurement channel at a different laser wavelength with a different RZPN code to measure the differential spectral absorption feature of the reflected laser light .The major technical challenge of RZPN lidars is the more sophisticated signal processing at the receiver. The receiver has to multiply the received signal with the kernel term by term and sum all the terms to obtain a single value of the cross-correlation function. The computation has to be repeated for every time shift of the cross-correlation function within the code length. Because of the narrow laser pulse width and the high signal sampling rate, the total number of computation steps required for a RZPN lidar can be difficult to carry out in real time. However, these computations are all simple and highly repetitive. The computations can be carried in parallel in hardware with available field programmable gate arrays (FPGAs). The correlation function can also be calculated in the frequency domain with the use of Fast Fourier Transform (FFT) and inverse FFT, which reduces the number of computations by orders of magnitude.Another limitation of RZPN lidars is the relatively long receiver integration time for a reliable measurement. During this time the target range needs to be constant, or the effect of the relative motion between the spacecraft and the target area needs to be properly compensated. This can be a challenge for measurements of major planets because of the relatively high orbit velocities of the spacecrafts. The Doppler shift due to the relative motion between the spacecraft and the target surface can be pre-compensated by adjusting the sampling clock frequency at the receiver based on the prediction. However, the small-scale surface topography cannot be predicted a priori, which can cause small time shifts in the received RZPN signal sequence and a reduction in the SNR. The residual changes in range rate in the orbit prediction can cause a time expansion or contraction in the received signal which causes a smear in the averaged signal and, in turn, the cross-correlation function. The smear can be reduced by shortening the signal averaging time, calculating the cross correlation at a higher rate, and averaging the correlation functions afterwards. However, this approach requires faster signal processing and more resources in the FPGA.RZPN lidar are best suited for missions to small planetary bodies, where the ground track speeds are relatively low and target range varies more slowly. We have carried out a detailed computer simulation to study the effects of range rate uncertainty, local topography, orbital effects, and the receiver comparator circuit on the correlation results. In addition, we have designed and implemented a real-time, FFT-based correlation breadboard using a Xilinx FPGA and the SALi RZPN code. These results will be reported in a future publication.A prototype SALi instrument is currently being developed using the RZPN lidar technique. It combines the designs of an existing CubeSat laser communication system ,49,50,51The SALi instrument uses a modular design. The primary focus of the current instrument development effort is the integration of the HgCdTe APD IDCA and real time RZPN signal processing with the FPGA. Other subsystem designs are relatively mature and they can be improved at a later time. For example, beryllium can be used instead of aluminum for the instrument chassis and support structure to improve the thermal conductance and reduce mass. A more efficient fiber laser assembly can be used to reduce the electrical power. A larger receiver telescope can also be used to extend the maximum measurement range.There are three instrument operation modes: the long-distance survey mode, the mid-altitude mapping mode, and the descent and landing mode. The parameter values of the RZPN code sequence are given in The most important aspects of SALi performance are the probability of target detection, the precision of ranging, and the precision of surface reflectance measurement. The ranging accuracy depends the frequency stability of the on-board clock and the requirement can be met with a proper choice of the clock oscillator and time keeping between the instrument and the spacecraft.The target is detected from a peak search of the correlation function. Because they contain a large number of signal and background photon counts summed over a relatively long integration time the peak and baseline of the correlation function can be approximated as Gaussian random variables according to the Central Limit Theorem. A matched filter can be used prior to the peak search to improve the SNR. The probability of correct detection can be written asandwidth , which iThe mean and variance of the peak of the correlation function with an ideal photon counting detector can be approximated asThe mean value of the baseline of the correlation function can be assumed to be zero for relatively long PN code. The variance of the baseline when the received laser pulses line up with the kernel in pulse intervals but not the bit pattern is given byThe variance when the received laser pulses fall within the return-to-zero portion of the kernel becomesTo simplify the calculation, we can use the worse-case variance given in Equation (12) for the entire baseline.The ranging precision can be estimated as in conventional lidar , as(15)The design of the laser transmitter and the optics are nearly identical to those used in the Compact Laser Communication Terminal (CLCT) for free-space laser communication demonstrations ,51. The The laser transmitter consists of a seed laser and an erbium dopped fiber amplifier (EDFA) at 1.55 μm wavelength. It was originally designed for transmitting a 64-ary pulse position modulation (PPM) signal at up to 6-W average output power . The lasThe receiver telescope, shown in In the current SALi design, the focus of the telescope and aft optics is fixed to an intermediate distance such that it can give a near optimal performance when mapping of the asteroid or the comet core but still acceptable performance when becoming out of focus at longer target range. During the spacecraft touch-and-go (TAG) maneuver at close up distance, the out of focus can become so severe that the IFOVs subtended by all pixels overlap and SALi becomes a single beam laser range finder. A focusing mechanism may be added in the aft optics assembly if a multipixel measurement is needed.The SALi detector assembly consists of a 2 × 8 pixel HgCdTe APD array ,59 in a The voltage outputs of the HgCdTe APD array are linear with the incident optical signal power. When there are multiple photons in a received laser pulse, the output becomes the linear sum of pulse waveforms from individual detected photons without any dead-time or other nonlinear effects. The APD has a factor of 100 linear dynamic range at a fixed APD gain. The APD gain can be adjusted by a factor of 1000. Together with the laser power adjustment, SALi can have at least six orders of magnitude dynamic range.In the present detectors the pixels are 20 μm in diameter and have a 64 μm pitch in a 2 × 8 format. Larger pixel arrays with more pixels are possible in the future. The HgCdTe APD arrays have been tested for radiation damage and shown to be suitable for use in a typical planetary mission . The useThe HgCdTe APD array operates at 110 K and a cryo-cooler is required. The cooler used in the current IDCA is a military off-the-shelf miniature Stirling cooler . It is dAn IDCA with the same HgCdTe APD array and the cooler has already been developed for a demonstration on a CubeSat . A microA major technical challenge for SALi is the receiver electronics that can process multi-channel RZPN signals in real time. Our previous RZPN lidar demonstrations post-processed the signal and kernel in software ,38, whicA clock generation circuit is added that generates the clock for the transmitted RZPN code and the receiver signal sampling clock. The clock for the received signal is locked to the transmitter clock but with an offset to compensate for the Doppler shift due to the relative motion between the spacecraft and the target body. The Doppler shift is estimated based on the predicted orbit dynamics. Both clocks are locked to an external reference clock, which is currently a temperature compensated crystal oscillator (TCXO) but can be replaced with a chip scale atomic clock (CSAC) , if needThe pulse waveforms from each of the 2 × 8 pixels are digitized before being processed by the FPGA. To reduce the electrical power consumption, a bank of comparators is used as 1-bit digitizers instead of regular 8-bit analog-to-digital converters (ADCs). The threshold levels for the comparators are dynamically adjusted according to the signal level. A single separate ADC is used to digitize the sum of all detector outputs to monitor the received signal level and control the comparator threshold, the laser power, and the detector gain. For weak signals, the detector gain is set to the highest and the thresholds are lowered to just above the noise floor such that the detector is in single photon counting mode. For strong signals, the detector gain is reduced and the thresholds are raised such that only multi-photon signal pulses are likely to be detected and background and detector dark noise are suppressed. The use of comparators causes quantization errors for strong signals, but the losses in SNR are more than compensated by the stronger signal and lower noise count rate.The FPGA is used to generate the RZPN code to modulate the laser transmitter and process the received signal to locate the target and calculate the laser pulse time of flight and average received laser power.The PN code is generated using the standard technique with a series of shift registers and logic feedbacks . The pulThe received signal from each pixel is serially fed into a shift register bank which periodically shifts out the content to an accumulator (histogrammer), which accumulates the signal over the RZPN code period. The range bin size of the histogrammer is the same as the signal sampling interval and the integration time of the histogrammer is set to 0.1 s, or 1538 codes. The cross correlation of the accumulated signal with the kernel is carried out in the frequency domain using FFTs, complex multiplication, and IFFTs. A peak search is conducted to locate the target return and determine the ambiguous range, return pulse width and energy. There are 16 sets of signal accumulation circuits, one for each pixel. The cross-correlation is carried out in series using one set of FFT/IFFT circuits. For the long-distance survey operation, the cross-correlation functions from all 16 pixels are summed together and then averaged up to 10 s integration time before the peak search. For short distance measurement during descent and landing, the laser is modulated with periodic pulses at about 65 kHz instead of a RZPN code. The received signals are sampled and histogrammed the same way but the FFT/IFFT is skipped and the signal goes directly to the peak search.The FPGA also processes the ADC signal and estimates the received signal and noise power and dynamically control the comparator threshold with the use of a standard control loop. The system controller dynamically adjusts the laser power and the APD gain via a software control loop based on the target return signal power and the background power.We have implemented the receiver processing algorithm in a breadboard test system using National Instruments (NI) FPGA hardware and LabVIEW software. The RZPN code and kernel were generated in the FPGA and the signals were transmitted and received in a loopback fashion. The received signal was histogrammed and corelated with the kernel in frequency space using the FFT method described above. A detailed description of that work will be described in a separate publication. We have verified that the correlation operation for each pixel can be carried out in <1 ms for the RZPN code specified in A small all-range lidar (SALi) is described, which is specially designed for space missions to small planetary bodies. The lidar uses return-to-zero pseudo-noise (RZPN) code laser modulation and correlation detection, which enables the use of low peak-power fiber lasers for long-distance ranging without aliasing. A newly developed, 2 × 8 pixel linear mode photon-counting HgCdTe avalanche photodiode (APD) array is used as the detector which gives a near quantum limited receiver sensitivity from 0.9 to 4.3 μm wavelengths. The laser power and the APD gain can both be adjusted to give six orders of magnitude dynamic range, enabling measurements of range and surface reflectance from hundreds of kilometers to near the target surface. The lidar is designed to operate in three modes, including reconnaissance at a long distance, mapping from orbit, and real-time multi-spot range and velocity measurements during descent and touchdown. A prototype instrument is currently being built by leveraging designs of the laser, optics, and electronics originally developed for a free-space laser communication demonstration. The combination of the fiber lasers, RZPN code modulation, and a multi-pixel single photon sensitive infrared detector enables a new class of small planetary lidars that are well suited for space missions to asteroids and comets.The development of the RZPN lidar technique resulted in a US Patent, 7,982,861 B2, 2011 , by Absh


With the increasing sophistication of the medical industry, various advanced medical services such as medical artificial intelligence, telemedicine, and personalized health care services have emerged. The demand for medical data is also rapidly increasing today because advanced medical services use medical data such as user data and electronic medical records (EMRs) to provide services. As a result, health care institutions and medical practitioners are researching various mechanisms and tools to feed medical data into their systems seamlessly. However, medical data contain sensitive personal information of patients. Therefore, ensuring security while meeting the demand for medical data is a very important problem in the information age for which a solution is required.Our goal is to design a blockchain-based decentralized patient information exchange (PIE) system that can safely and efficiently share EMRs. The proposed system preserves patients’ privacy in the EMRs through a medical information exchange process that includes data encryption and access control.We propose a blockchain-based EMR-sharing system that allows patients to manage their EMRs scattered across multiple hospitals and share them with other users. Our PIE system protects the patient’s EMR from security threats such as counterfeiting and privacy attacks during data sharing. In addition, it provides scalability by using distributed data-sharing methods to quickly share an EMR, regardless of its size or type. We implemented simulation models using Hyperledger Fabric, an open source blockchain framework.We performed a simulation of the EMR-sharing process and compared it with previous works on blockchain-based medical systems to check the proposed system’s performance. During the simulation, we found that it takes an average of 0.01014 (SD 0.0028) seconds to download 1 MB of EMR in our proposed PIE system. Moreover, it has been confirmed that data can be freely shared with other users regardless of the size or format of the data to be transmitted through the distributed data-sharing technique using the InterPlanetary File System. We conducted a security analysis to check whether the proposed security mechanism can effectively protect users of the EMR-sharing system from security threats such as data forgery or unauthorized access, and we found that the distributed ledger structure and re-encryption–based data encryption method can effectively protect users’ EMRs from forgery and privacy leak threats and provide data integrity.Blockchain is a distributed ledger technology that provides data integrity to enable patient-centered health information exchange and access control. PIE systems integrate and manage fragmented patient EMRs through blockchain and protect users from security threats during the data exchange process among users. To increase safety and efficiency in the EMR-sharing process, we used access control using security levels, data encryption based on re-encryption, and a distributed data-sharing scheme. With the development of information and communication technology, the existing medical information system, which used paper charts to manage medical information such as patient treatment information and clinical results, changed to a digital-based medical information system. As of 2017, more than 94% of the hospitals in the United States have used digital health information systems . MoreoveMedical information that directly affects a patient’s health must have integrity and be reliable. Moreover, the patient’s privacy should be protected from exposure to unauthorized users. Therefore, it is necessary to develop a secure EMR-sharing system that can provide the integrity and reliability of an EMR and protect patient privacy by addressing the problems of the existing centralized EMR-sharing systems. Decentralization of the system has been proposed to complement the problems of the existing EMR-sharing system, and blockchain is receiving much attention as a technology suited for this purpose ,28. BlocThe blockchain-based decentralized EMR-sharing system has the opposite characteristics to the existing client–server-based centralized system. Through these opposite characteristics, the blockchain-based EMR-sharing system overcomes the current system’s problems and provides various advantages. Unlike the existing centralized system, the blockchain-based decentralized EMR-sharing system exhibits strong resistance to the single point of failure because no central administrator or server controls the system. As multiple nodes operate the decentralized EMR system, data loss or service failure can be prevented even if a specific node fails. Therefore, it is possible to build a more robust system and provide stable service.In terms of performance, the blockchain-based EMR-sharing system exhibits low throughput and high latency compared with centralized systems because of the data propagation delay between nodes and the consensus mechanism. However, performance problems can be overcome through various methods, including automation of the system by using smart contracts, lightweight consensus mechanisms, and private blockchain models. In a centralized system, only the central administrator manages the database. Hence, the data stored in the database can be arbitrarily modified or deleted only by the central administrator. However, to modify stored data in the decentralized EMR-sharing system, the consent of most of the blockchain nodes is required; therefore, an arbitrary user cannot modify the data at will. Therefore, the blockchain-based EMR-sharing system provides high data integrity and a transparent process, allowing EMRs to be shared without the intervention of a third party, unlike centralized systems. The decentralized EMR-sharing system prevents data leakage and privacy threats from third parties with these characteristics.Despite many concerns about technological limitations and suitability, many researchers are studying blockchain-based EMR-sharing systems to take advantage of the benefits of blockchain technology -42. TablResearchers have proposed various EMR-sharing system models based on blockchain to secure the integrity and reliability of EMRs and build a secure EMR-sharing environment. The studies on EMR-sharing systems based on blockchain technology are presented in Azaria et al  proposedHylock and Zeng  proposedWu and Du  proposedAbdellatif et al  proposedConnecting Health and Care for the Nation: A Shared Nationwide Interoperability Roadmap – Version 1.0, prepared by the Office of the National Coordinator for Health Information Technology (ONC) [Existing studies on blockchain-based EMR-sharing systems have used blockchain models designed for cryptocurrencies such as Bitcoin and Ethereum. However, existing blockchain models for cryptocurrency have limitations in providing the security and scalability required in sharing EMRs. The system also failed to meet the requirements of EMR-sharing systems as defined in gy (ONC) . TherefoIn this paper, we propose a patient information exchange (PIE) system. The proposed blockchain-based EMR-sharing system overcomes the limitations of existing blockchain-based EMR-sharing systems and satisfies the privacy and security framework defined by the ONC. Furthermore, our proposed system prevents data loss and privacy breaches in sharing data through the data encryption scheme based on re-encryption, ensuring strong data security. Moreover, data integrity is ensured by preventing the forgery and alteration of EMRs by using the decentralized ledger structure and the unique hash value of the data. Furthermore, allowing patients to set their data-access rights ensures patient ownership of their EMR and establishes a patient-centered medical system. Moreover, the PIE system provides improved performance by solving the low processing performance and scalability issues due to the limited block capacity of the existing blockchain through the distributed data-sharing method using the IPFS. As a result, we contribute business process optimization, cost reduction, patient outcome improvements, and enhanced compliance in the health care field ,58.Components of the Proposed PIE System section, we define the entities that make up the components of the system and describe each entity’s role. In the EMR Transaction Structure section, we describe the structure and components designed to share EMRs effectively. Finally, in the Security Levels of EMRs section, we discuss the security level, which depends on the EMR data type and classifies the data based on the type. The system model of the proposed PIE system is shown in Here, we describe the proposed PIE system. In the We propose a blockchain-based PIE system to improve the security and efficiency of the EMR-sharing process. To prevent forgery of EMRs and protect patient privacy, we use a consortium blockchain model in which only authorized users can participate. The medical consortium that operates and manages the blockchain comprises state-approved and trusted medical institutions. As the proposed blockchain-based PIE system uses a private blockchain model, the consensus algorithm in the block generation process is not addressed. Instead, the chain is constructed by sequentially storing the generated EMR transactions to create a block and connecting them. Hospitals and medical institutions serve as blockchain nodes that issue EMR transactions and store them in block form. Health care workers and patients who create and use EMRs participate in the blockchain network as users by using IDs issued according to user type after a certification process by a CA. Users participating in the blockchain network can register their EMRs on the blockchain and use them at any time. The proposed PIE system is a patient-centered EMR-sharing system where patients directly participate in the EMR upload and EMR-sharing process. The patient directly generates a key to encrypt the EMR and defines the categories of users who can access the EMR. By allowing patients to manage their own EMRs, we build a user-centered system that protects patients’ privacy and gives them ownership of their EMRs.The proposed PIE system securely protects patient EMRs from security threats such as data forgery and personal information leakage, which can occur during EMR management and sharing. To protect EMRs from the aforementioned security threats, we use public key–based asymmetric encryption and our proposed PRE-based decryption authority delegation mechanism. The proposed decryption authority delegation mechanism prevents private key leakage by decrypting data encrypted with the public key. Moreover, delegating authority to decrypt data solves the problem of data access in an emergency when a patient cannot respond to a request for access to their EMR, as de Oliveira et al  suggesteThe performance and scalability of the PIE system are enhanced by using the IPFS, which supports distributed data-sharing technology. The EMR encrypted with the patient’s encryption key is stored on the IPFS network, and the hash value of the EMR is stored on the medical blockchain in the form of metadata. Instead of storing the data as a whole in the blockchain, it is possible to reduce the load on the system by storing only the hash value of the data. Furthermore, if data are shared using the IPFS, large-capacity data such as magnetic resonance imaging, computed tomography, and endoscopy images can also be shared, improving the scalability of the blockchain system.The proposed PIE system consists of blockchain nodes , users of the blockchain network , and the IPFS. The role of each entity is outlined in the following paragraph:A medical consortium consisting of hospitals and medical institutions that wish to share EMRs builds and manages a distributed ledger as operator of a permissioned blockchain network in which only authorized users can participate. The medical consortium blockchain stores the information of the EMRs generated by each hospital. The information recorded on the blockchain is a hash value of real medical data stored in the IPFS and simplified medical information that users can comprehend. Data registered on the blockchain cannot be arbitrarily deleted or modified, providing high reliability and medical data integrity. Patients and physicians, who are the users of the blockchain network, share EMR information through the network. Patients can use a decentralized app to share their EMRs in the PIE system. Furthermore, patients set their APs for their EMRs and generate re-encryption keys for re-encryption. Unlike traditional hospital-centered health care systems, the PIE system guarantees the patient’s ownership of their EMR. In a patient-centered health care system, where patients have rights to their own EMRs, they have the freedom to choose who can use their EMR and their data at any time. Furthermore, patients may sell their medical data to research institutions or hospitals, in addition to using the data for therapeutic purposes. Health care workers consist of reliable physicians and health care service providers such as medical researchers and insurance agents. Health care workers use computer systems at hospitals or medical institutions to encrypt EMRs generated during the patient treatment process and upload them to the IPFS. After uploading the EMRs, health care workers submit the EMR information to their hospitals and institutions. Health care workers also serve as consumers of medical data by, for example, sharing EMRs through a blockchain network to treat patients or using the data for clinical research. The IPFS is a distributed file-sharing system that splits data stored on multiple computers worldwide into small pieces and shares only a portion. The distributed data-sharing method used by the IPFS enables rapid sharing of large-capacity data such as magnetic resonance imaging or computed tomography images. In addition, the IPFS prevents duplicate creation and storage of medical data by managing data with hash values based on data content.*qx, y ∈ Z, it is difficult for any probabilistic polynomial time attacker A to find a value m *q∈ Z such that x = my. Therefore, the attacker cannot obtain the private key from a public key or ciphertext. Our system model considers external threats from outside the system and internal threats from the system participants. We assume that both threats are in the form of a logical attack, not a physical attack. The external threats target the patients’ private data such as the EMR, insurance details, and other personal information. For example, an external attacker wants to eavesdrop on all communication among the participants to obtain patients’ personal information. Internal attackers can include health care researchers or insurance agents. They are allowed access to limited information in the form of an abstract regarding disease and length of hospital stay, not details of the disease or patients’ personal information. However, internal attackers are snooping for patients’ private data; therefore, they try to access their medical information. In addition, internal attackers attempt to manipulate clinical results or commit insurance fraud by arbitrarily forging a patient’s EMR. In this study, we consider the traditional cryptographic system, not the postquantum cryptographic system. Therefore, we use the discrete logarithm problem, which is one of the difficult problems of 1-way functions. The discrete logarithm problem is one where given The proposed blockchain-based medical system uses transactions designed to effectively share the desired medical data while preventing leakage of personal information and data when uploading the medical data to the blockchain. A unique identifier or ID is used in the blockchain network by the physician who created the EMR and the patient who is the owner of the generated data. The CA issues a user ID according to the type of user participating in the blockchain network. A user ID is a randomly generated value consisting of numbers and letters; it is possible to identify users but not know who the owner is. Information about users who can map users to user IDs is securely managed by a CA such as the trusted government authority that issued the ID. As the user IDs are correlated, users are protected from the threat of personal information leakage ,61. The Patient EMRs may include data relating to clinical trials and insurance as well as sensor data generated by health care devices, in addition to medical information generated during the process of receiving treatment in hospitals. Depending on the EMR data type, the required security levels will differ. For example, if information such as name, residence, and social security number, which can identify an individual, is leaked to outside parties, it can lead to serious personal information leakage; consequently, a high security level is required. Conversely, information that is not personally identifiable, such as gender, age, eating habits, and exercise status, does not require a high security level because it is not a serious problem even if this information is disclosed to outside parties. Therefore, it is necessary to provide differentiated security levels and separate management, with respect to the sensitivity of the personal information, according to the EMR data type.The minimum security level required for each data type is established by categorizing privacy sensitivity according to data type and evaluating accessibility and data potential per user. The security levels assigned according to the sensitivity of private information fall into three classes: private, moderate, and low. This section presents the EMR-sharing process that protects the patient’s EMR from various attacks and safely shares it. Moreover, it describes the work performed in each process. A more detailed description of each process-specific algorithm is provided in *qx ∈ Z. The selected x is set as the user’s secret key and is never shared. Next, users generate a public key for use in the network using their private key and the key generator. Users who can directly generate the encryption key that they use for EMR sharing can generate an encryption key each time an EMR is created, thus protecting the EMR with a different key each time. Because of the nature of an EMR that has been used for a long time, the encryption key must be maintained as is when data are encrypted [Patients and physicians want to participate in the blockchain network to manage EMRs and securely share them with other users. Users participate in the blockchain network through the user registration process, which consists of an identity registration phase to register the user’s identity and an authentication phase to obtain security parameters to generate an encryption key. ncrypted . HoweverPDiRK to re-encrypt the encrypted EMR. The re-encryption key is generated using the patient’s private key and the physician’s public key. At the same time, the patient creates an AP that defines the level of users who can access the EMR. Next, the patient sends the dedicated encryption key PDiRK, the AP, and the user ID, which was created by the patient, to the physician who provided treatment. Upon receiving the message from the patient, the physician encrypts the patient’s EMR with the patient’s dedicated encryption key. The physician then uploads the encrypted EMR, PC, to the IPFS and receives the hash value, P)hashhash, and access policyInput: Secret keyOutput: Re-encryption key, access policy, summary information from electronic medical record, hash value returned by the InterPlanetary File System, and transactionr to generate a dedicated encryption key for encrypting their electronic medical record.The patient selects a random security parameter value PSK and the physician’s public key DPK to generate a re-encryption key P→DiRK for re-encrypting their PEMR.The patient uses their secret key The patient generates an access policy that defines which users can access their electronic medical record.P→DiRK, access policy, and PUserID to the physician who treated them.The patient transmits the dedicated encryption key, P→Di, access policy, and PUserID from the patient encrypts the patient’s medical record EMRP using the dedicated encryption key.The physician who receives the dedicated encryption key, RKPC to the InterPlanetary File System and receives the p)hash to quickly search for a transaction containing the information of the desired EMR. Then, the requester downloads the encrypted EMR from the IPFS using the acquired transaction information. As the downloaded EMR is encrypted with a dedicated encryption key, it must be decrypted using the patient’s private key or re-encrypted using the re-encryption key before using it. However, because sharing the patient’s private key is very dangerous, the requester must send a message to the patient requesting a re-encryption key for re-encryption. The message requesting the re-encryption key goes through the user verification process of the access check contract. Initially, the user verification process checks whether the requester’s security level satisfies the patient’s EMR AP. When it is confirmed that the user has met the required security level, a message requesting a re-encryption key is sent to the patient, the owner of the EMR. Upon receiving the message requesting the re-encryption key, the patient sends Algorithm 2Input: Summary information from the electronic medical record, user ID, and user’s security leveldCand P→RequesterRKOutput: Re-encrypted electronic medical record The physician executes a smart contract for electronic medical record retrieval to find a transaction containing the information of the desired electronic medical record.The smart contract uses the user ID or the electronic medical record’s summarized information to find a transaction containing the desired information and returns it.PC from the InterPlanetary File System using the hash value contained in the transaction.The physician who receives the transaction information downloads the encrypted electronic medical record EMRPRK for re-encryption from the patient who is the owner of the encrypted electronic medical record.The physician executes the re-encryption key request smart contract to request the key The smart contract for requesting the re-encryption key performs a user authentication step to verify that the security level submitted by the user who requested the message transmission satisfies the access policy set by the patient.If the security level of the user requesting the re-encryption key satisfies the access policy, the smart contract sends a message requesting the re-encryption key to the patient .P→RequesterRK using the requester’s public key and send it to the requester.The patient who receives the message requesting the re-encryption key generates P→RequesterRK uses it to re-encrypt PC into DC.The physician receiving DC to obtain the original PEMR.The physician uses their private key to decrypt A simulation was designed to verify that the proposed blockchain-based medical system sufficiently reflects the medical field’s requirements and enables safe data sharing. We simulated the process of sharing encrypted EMRs over a blockchain network. In the EMR-sharing process of uploading and downloading EMRs over the network, we checked the effect of the time taken to process the data and the size of the data shared on the system’s performance. Moreover, the performance of the proposed smart contract–based re-encryption key–sharing method was verified by measuring the execution time of the smart contract. This paper does not cover improvements to the consensus process performed on the blockchain network; therefore, the improvements were not evaluated.Methods section. The simulation consisted of 2 physicians, a patient, a medical consortium, and a public IPFS network participating in the blockchain network as the minimum unit for sharing an EMR. Entities on the network were classified into two categories: a host operating a blockchain network and a guest operating on the network. All processes in the proposed system can be viewed as interactions among guests. In the test environment, there was a consortium to which many hospitals belonged, and each hospital had a Flask server with a different port. The code of the software development kit used to implement the blockchain network is Node.js. The host computer was a PC running Windows 10 Pro 64x (Microsoft Corp) in a wired environment, with 32 GB memory and Intel Core i7-10700K central processing unit at 3.80 GHz. The guest computer was a PC running Ubuntu 18.0.4 64x  in a wired environment, with 10 GB memory and Intel Core i7-10700K central processing unit at 3.80 GHz. The communication rates considered were the download and upload rates in megabits per second: host upload and download rates and guest upload and download rates.The test environment was designed based on the data-sharing process defined in the The simulation was performed on 1 PC to directly compare the host’s and guest’s processing times. Blockchain implementation was performed using Hyperledger 2.3.1 ; Apache To evaluate the performance of the proposed PIE system, we measured the time required for the EMR-sharing process and the execution time of the smart contract for re-encryption key sharing. The EMR-sharing process was divided into upload and download processes, and the time taken to perform each process was measured. The execution time of the EMR upload process was defined as the time taken to upload the EMR to the IPFS and post the returned EMR hash value to the blockchain. The execution time of the EMR download process was a measure of how long it took users to download the EMR over the IPFS. Considering the characteristics of an EMR that supports various types of data, the simulation was performed using various data, ranging from text format (0.4 kB) to medical images (1 GB). The simulation measured only the time required in the communication process for exchanging data among users and did not consider the impact on the process of the data encryption and decryption operations. To objectively evaluate the performance of the proposed system, we performed a comparative analysis with existing blockchain-based medical information exchange systems. Simulations were performed for three types of systems ; the results of the simulations for the EMR upload process are shown in Through the EMR upload simulation it was confirmed that the larger the data to be uploaded, the longer it takes; the larger the amount of data to be uploaded, the higher the required data rate. As a result, the processing time increased dramatically for data that exceeded the acceptable data rate (100 Mbps) in the simulation environment. In an on-chain–based system that stores data in the original form in blocks, the size of data that can be uploaded is limited to 1 MB, which is the maximum size of the block; therefore, there is no simulation result for data beyond that size. Most of the time taken to upload an EMR was when a query request needed to be made to the blockchain network, which took an average of 2.1 (SD 0.0343) seconds. The actual time taken to upload an EMR to the IPFS increased depending on the size of the data, but it was very short. A graph of the time it takes to upload EMR to IPFS can be found in The simulation results for downloading the EMR posted on the blockchain are presented in We performed a simulation of the smart contract–based re-encryption key–sharing process. The re-encryption key–sharing process verifies the user requesting the re-encryption key to use the patient’s EMR and passes the re-encryption key to the user. The user who receives the re-encryption key performs the re-encryption process and finally decrypts the encrypted EMR using their private key to use the patient’s EMR. A graph of the smart contract simulation results for re-encryption key sharing is in In this section, we will check how the proposed PIE system effectively responds to security threats and analyze whether it is possible to share secure medical data using the proposed PIE system.RKrequestSmart contract verifies the requester’s security level and accepts or declines the request depending on the AP set by the patient. The access control scheme using a smart contract can protect the patient’s privacy from internal threats.When medical data are shared using a network, an external attacker can obtain the medical data through a sniffing or eavesdropping attack. If medical data are leaked, the patient’s privacy in the EMR is also exposed. In the proposed PIE system, medical data are encrypted using the dedicated encryption key for the safe sharing of medical data. As encrypted medical data can only be decrypted by the patient or by a user approved by the patient, the information in the data is not exposed even if the data are stolen. By granting data decryption authority using the PRE technique, the user approved by the patient can decrypt the data using their private key during the data decryption step. The proposed EMR-sharing method prevents the leakage of private information during the EMR-sharing process and ensures safety by eliminating the private key–exchange process for data decryption. If an internal attacker attempts unauthorized access to the patient’s information, in our proposed system, The internal attacker can perform forgery attacks by accessing the medical data that medical institutions manage independently. If the original data stored at a medical institution are damaged, it is difficult to recover the data; moreover, it is also significantly challenging to determine whether the data have been forged or altered. These attacks can be effectively prevented by storing and managing EMR-related information such as the hash value of medical data, publicly available medical information, and hospital ID in the blockchain. As the EMR information recorded on the blockchain contains the information at the time of creation, it is easy to check whether the data are damaged. If the data are damaged, they can be quickly restored using the distributed data-sharing method. For an attacker to forge the data stored in the blockchain, they must possess the mighty hash computation power of more than 50% of the entire network and create new blocks faster than other honest nodes propagate them to the network. As meeting the necessary conditions to forge blockchain data is challenging, attackers cannot delete or modify data. Therefore, using a blockchain-based medical system ensures medical data integrity and reliability, thereby enabling safe medical data management and sharing.The external attacker can perform denial-of-service attacks on the PIE system. As a result, the system’s operation becomes abnormal and it produces unexpected outputs. The system we propose is directly or indirectly related to patients’ lives; therefore, high availability is important. Hence, we use distributed systems such as a medical consortium blockchain and the IPFS. If the attacker breaks down the sharing system, patients cannot share their medical data and physicians or health care providers cannot obtain the required information. However, in the proposed sharing scheme based on blockchain, if the attacker makes a few of the blockchain nodes unavailable, the other nodes can provide the needed services.The study’s principal findings concern implementing integrated management of fragmented EMRs, preventing leakage of personal information of patients during the EMR-sharing process, and establishing a patient-centered medical data system by granting decryption authority, as outlined in the following list:We designed a blockchain-based PIE medical system that effectively manages and shares medical data. EMRs generated by different medical institutions are managed through a blockchain network to prevent the fragmentation of medical data. Moreover, through the PIE system, duplicate EMRs can be avoided, reducing the cost and wastage of storage space.The PIE system encrypts the patient’s medical data and uploads and shares the encrypted EMR and data identification parameters to the network with minimal medical information. Thus, the proposed method fundamentally overcomes the problem of possible leakage of personal data when the data are posted on the blockchain for sharing with other network members. Therefore, privacy preservation required in a system handling sensitive EMR information is guaranteed, making safe EMR management and sharing possible.Our system reinforces the patient’s role in the medical system by allowing them to grant decryption rights to their data to other users using re-encryption techniques. If other users  wish to use a patient’s EMR data, they must obtain a re-encryption key and re-encrypt the EMR data. Building a patient-centered medical data system differs from the existing hospital-centered medical data system in that the patient’s role is reinforced in our system.Blockchain-based medical systems receive considerable attention as next-generation medical systems that will replace existing medical data management systems, and numerous researchers are conducting various studies. However, blockchain-based medical systems’ technological maturity remains at the prototype level. Moreover, as the medical data formats used by different countries or institutions vary considerably, it is challenging to share the medical data. Consequently, research into standardized medical data formats such as the Health Level 7 Fast Healthcare Interoperability Resources  is requiThis paper presented the PIE system based on a consortium blockchain that allows patients to manage their medical data. The PIE system can securely manage and share EMRs by overcoming the existing blockchain-based medical system’s problems. The PIE system uses a distributed data-sharing method and lightweight transaction structure to solve scalability and privacy issues, a chronic problem of blockchain-based medical systems. By rapidly sharing large-capacity data such as medical images using a distributed data-sharing method, the issues of low processing speed and block sizes of existing blockchains are addressed. Lightweight transactions can store more information in blocks because they contain only minimal information, such as the encrypted EMR metadata and EMR summary information. The vast amount of medical data generated daily is effectively processed and managed using a lightweight transaction structure. The re-encryption–based data encryption method is used to resolve the problem of leakage of data and personal information when sharing EMRs. Even if the EMR encrypted with the dedicated encryption key is leaked during the sharing process, it cannot be decrypted; therefore, it is safe from the threat of leakage. Honest users wishing to use the patient’s data can re-encrypt the EMR by obtaining a re-encryption key from the patient. The EMR-sharing process was performed using smart contracts. Security level–based access control was performed using smart contracts to prevent unauthorized users from using medical data, and re-encryption keys were delivered only to authorized users. As a result, the proposed blockchain-based medical system provides improved security and scalability, enabling efficient and safe medical data sharing.


Collaboration is an important 21st Century skill. Co-located (or face-to-face) collaboration (CC) analytics gained momentum with the advent of sensor technology. Most of these works have used the audio modality to detect the quality of CC. The CC quality can be detected from simple indicators of collaboration such as total speaking time or complex indicators like synchrony in the rise and fall of the average pitch. Most studies in the past focused on “how group members talk”  and not “what they talk”. The “what” of the conversations is more overt contrary to the “how” of the conversations. Very few studies studied “what” group members talk about, and these studies were lab based showing a representative overview of specific words as topic clusters instead of analysing the richness of the content of the conversations by understanding the linkage between these words. To overcome this, we made a starting step in this technical paper based on field trials to prototype a tool to move towards automatic collaboration analytics. We designed a technical setup to collect, process and visualize audio data automatically. The data collection took place while a board game was played among the university staff with pre-assigned roles to create awareness of the connection between learning analytics and learning design. We not only did a word-level analysis of the conversations, but also analysed the richness of these conversations by visualizing the strength of the linkage between these words and phrases interactively. In this visualization, we used a network graph to visualize turn taking exchange between different roles along with the word-level and phrase-level analysis. We also used centrality measures to understand the network graph further based on how much words have hold over the network of words and how influential are certain words. Finally, we found that this approach had certain limitations in terms of automation in speaker diarization  and text data pre-processing. Therefore, we concluded that even though the technical setup was partially automated, it is a way forward to understand the richness of the conversations between different roles and makes a significant step towards automatic collaboration analytics. It involves discourse, gesture, gaze, cognition, social skills, tacit practices, etc.” , the behaviour of team members , the types of interaction  and behaviour during collaboration .The quality of co-located collaboration from group speech data alone has been detected in the past using indicators of collaboration derived from audio. The indicators of collaboration can be as simple as total speaking time of a group member  or as conon-verbal indicators like total speaking time [Most studies on CC in the past focused on automated analysis using temporal , spectral indicators of speech  ,20 and oing time ,18, freqing time  or usinging time ,22. Thering time ,20.non-verbal audio indicators, very few studies used the verbal audio indicators or the content of the audio for the analysis of CC quality. For example, for “talk traces” [RQ:To what extent can co-located collaboration analytics from group speech data be done automatically?Apart from the majority of studies focusing on the analysis of  traces”  and “mee traces” , verbal  traces”  did topi traces”  classifiRQ1:What co-located collaboration indicators have been detected from group speech data in past studies?RQ2:What collaboration analytics can be employed to analyse group speech data from co-located collaboration automatically?To answer this primary research question, we sub-divided it into two sub-research questions:RQ1, we look at the already available literature in RQ2, we designed a technical setup and report about the materials and methods used in To answer team composition , the behaviour of team members , the types of interaction  and behaviour during collaboration .Audio is a commonly occurring modality during collaboration ,17,18,19speaking time and eye gaze behaviour. Feedback was given in the form of different coloured circles representing attention to and from speakers and listeners measured by eye gaze and the total speaking time of that member. This feedback was projected on top of the table in front of where each participant was sitting using a top-down projector. They performed both quantitative and qualitative evaluation of the effect of the feedback: the feedback was accepted as a positive measure by most group members; the use of feedback had a positive impact on the behaviour of group members as they had a balanced participation. There was a balanced participation in terms of the speaking time of each group member. It was found that the eye gaze measured to track the total attention of the listener and speaker was not a good predictor of the quality of collaboration. As per the authors, this was because of the difficulty in intuitively controlling gaze behaviour as compared to controlling the speaking behaviour even though both can be consciously controlled.To elaborate further, Terken and Strum  designedSome other works also used total speaking time as an indicator of collaboration ,18. The group speech activity indicators  and individual speaker-based indicators  were good predictors of collaboration quality as marked by the annotators. Moreover, the group-level indicators alone were good predictors of collaboration quality. According to the authors, this was because the individual speaker-based indicators were agnostic to the group information, contrary to the group speech activity indicators. All these indicators were fed to a machine learning classifier to get the measurements, so in the end, it was a black-box approach. They did not employ any fine-grained analysis, which could help to uncover the degree of contribution of different indicators to the prediction of good or bad collaboration quality.To analyse other audio indicators in depth, Bassiou et al.  used nonspeaker-based indicators like the intensity, pitch and jitter were used to detect collaboration quality among working pairs [turn taking) with overlap of speech [Similarly, ng pairs . When twng pairs . Lubold ng pairs  found a ng pairs . The pref speech  indicatef speech ,30.overlapped speech was an indicator of constructive problem-solving progress, expertise and collaboration. They used both the number of overlaps in speech and the duration of overlap in speech when tracking the interruptions during speaking. Luz and Saturnino [Additionally, other works focused on expertise detection and productive problem solving ,25,31, eaturnino  used theaturnino  used audAll the above studies analysed the non-verbal audio indicators  instead of the verbal audio indicators of collaboration. Non-verbal audio indicators of collaboration are less overt as compared to verbal audio indicators. Semantically, the content of the conversation, i.e., the verbal audio indicators of collaboration, have the same meaning most of the times.With the rise of automatic speech recognition techniques, few studies   game. It is a board game [https://game.fola2.com/, last accessed on 30 April 2021) currently under development) played face-to-face with different themed cards and roles that is used in workshops to create awareness of the connection between learning analytics and the learning design. It also can be used as an instrument to collect indicators when planning learning analytics already while designing learning activities. This game was used in 14 face-to-face meetings (with each meeting spanning between 60 and 90 min) among different teaching staff and other staff of a university. This task had different phases, which were colour-coded based on the cards supposed to be used in that phase  (the phases and cards have the same meaning and are being used interchangeably henceforth) with different roles assigned to each member. The blue (card) phase defines the steps in the learning activity. Each learning activity consists of a sequence of interactions such as learner to teacher, learner with learning environment, material to learner and so on. The red phase or learning enhancing technology cards are part of the step in the game where we search for enhancements of the interactions using technology such as sensors, virtual reality, etc. The yellow phase defines what we want to know about the interaction or within the learning activity. For example, it can be engagement, social interaction or how students take initiative. The yellow cards can be used to get input on what teachers do in classrooms to value their design choices or actions. Each card also had some prompts to steer the group conversation.The collaboration task that we used as the basis for our audio recordings was to design a learning activity using the Fellowship of Learning Activity ((FOLA)ard game   advisor, game master, educational advisor and teacher. These roles had the same meaning as a real-life student, teacher or advisor, while the game master was the main moderator of the game who also helped to steer the conversation during the task. Each group member had a clip-on microphone attached along with the respective audio recorder, which recorded and stored the conversation locally in that recorder. Next, we outline the architecture used for data collection, processing and analysis.We recorded the conversations during these meetings. The conversations were in Dutch. Each group member was pre-assigned roles during the conversation: First, the audio files from each group member were saved into a storage space in the respective local device, i.e., the audio recorder. Then, after the meeting, these files were immediately transferred to the central storage space, which was the long-term storage. For the pre-processing and subsequent operations on the data, we did not disturb the original data collected, rather we took a copy of the files in the storage space for the pre-processing and processing unit. Here, we pre-processed and transcribed these audio files using Google speech-to-text. Finally, the data were processed and analysed to generate meaningful insights and passed on to the visualization unit to generate the visualizations. These visualizations were generated in a post hoc manner after the group meetings. VoxSort Diarization  software. This is known as speaker diarization, which helped us figure out “who spoke when?”. The group members playing specific roles were anonymized by mapping their names to roles, and these audio files were combined into one file. Finally, we transcribed this audio file using Google speech-to-text (free version) in Python. For this transcription, we split the audio file into 5 s window chunks , which worked the best in our case. This helped us to increase the accuracy of the transcription, which was otherwise not good enough when we used long files of 1 to 30 min in duration. This process was repeated, and finally, these chunks were combined, sorted by timestamps from the beginning till the end of the meeting. While performing these subsequent steps, the information like timestamp and speaker role was put into the data tables in a .csv file format. The data pre-processing, processing, analysis and visualizations were done in Python using different openly available libraries. We pre-processed the stored audio files for each group member by extracting the timestamp of the audio file, which denoted the exact end time of that audio file (in .wav audio file format) and the duration of the audio file. Then, we derived the start time of the audio file using the end time and the duration of the file. Next, we associated the conversations in each audio file with the group member playing a certain role, thereby associating with the group member who was speaking using start time, the end time, the roles of the group member under names (256 under names denotes noise) and the utterance as text_y. This data table represented the ordered conversation from the beginning till the end of the meeting. If Google speech-to-text failed to transcribe an audio file, then the corresponding text entry was left blank. Normally, this happened when a part of the audio was of a really short duration or had some random sounds like a click sound, um, claps or laughter. Now, the wrangled  data were ready for processing and analysis.Then, the final extracted data along with the timestamps were stored in data tables in a .csv file format, as shown in the natural language processing. The usual approach in natural language processing is to first to clean the text. Next, we built the text model from the conversation corpus. We had to make sure that our text model could understand similarities and also understand when two different words meant similar things. Therefore, the following steps were taken by us in order to achieve this cleaning:Tokenization—The process of splitting sentences into good words or tokens. It lays the foundation for the next steps of cleansing.Elimination of stop words—The process of removing words that mean little; these are usually words that occur very frequently. Apart from using the libraries in Python for stop word removal, we also defined our list of contextual stop words that were considered unimportant for this model.Lemmatization and stemming—Lemmatization and stemming convert a word into its root form. For example, for the words running and runs, the stem of both words is run. Thus, after we stemmed, these words would be grouped together and retain same meaning for the model even though they had different forms.Sentence segmentation—We split the unstructured spoken text into different sentences, which helped the model understand the boundaries of the long text to make it more semantically distinct.Vectorization—Since we cannot input plain words into a model and expect it to learn from it, we had to vectorize the words. This basically means creating unit vectors for all words. As the machine can understand numbers only, so the vectorized version of words will create a dictionary for the model, which would be useful later while generating bigrams (two word combinations appearing together), trigrams (three word combinations appearing together) and topic modelling based on the keywords.The data table stored as a .csv file was processed. Our primary focus was to analyse the content of the conversations to generate meaningful insights. To do this, we proceeded with text cleaning, processing and analysis, which come under the umbrella term of After cleaning, we proceeded with the analysis and visualizations. For the analysis, we first visualized an exploratory view of the keywords used in different phases of the CC task by different roles with the help of the cleaned text model to understand the content of the conversations. Then, we proceeded with the detailed analysis with the help of richer visualizations to understand the content and context of the conversation further with the help of our technical setup. In the subsequent sub-section, we describe the materials and methods used for the data analysis.For the scope of this article, where we describe a proof-of-concept to prototype the development of our technical setup towards automatic collaboration analytics for group speech data, we restricted our analysis to only the first out of the 14 meetings. First, we visualized in an exploratory manner to see the frequently used keywords in the text model by different group members playing different roles by using frequency analysis. To make sense of the visualizations and understand the context of the conversations, we took the help of the game master to generate summarized annotations for each phase in English  and LSI (latent semantic indexing) in different phases of the meeting session. LSI helped us to identify the coherence score based on which we decided the ideal number of topics in that phase, and then, we used LDA (which is a probabilistic approach of topic modelling) in multiple iterations to find these topical clusters. LDA and LSI just show the representative keyword collection and are unsupervised algorithms for topic modelling, which can cluster semantically similar words without the need for user labelling (or input).To go in depth into the representative overview of these words in relation to the word exchange between different roles, we looked at the different bigrams (consecutive two-word phrases) and also ranked them based on the tf-idf (term frequency-inverse document frequency) ranking. tf-idf ranking of the bigrams helped to give an overview about the frequently (with a lower tf-idf ranking) and rarely (with a higher tf-idf ranking) used bigrams. Next, we wanted to see the relationship between these words and phrases with the help of parts-of-speech tagging and construction of knowledge graphs. Knowledge graphs show the relationship between the subject, the object with the verb or the verb phrase linking them.Knowledge graphs  are sometimes difficult to interpret because of the inaccurate sentence segmentation of unstructured data. Moreover, they also do not show the strength of different words  and the strength of linkage between these words. To show this, a co-occurrence matrix was made. A co-occurrence matrix shows how many times the words co-occur in the same sentence. For example, in the two sentences: “I love riding bike” and “Bike ride is loved by many”, the co-occurrence matrix after doing text pre-processing  would be as in Then, we visualized this co-occurrence matrix using social network analysis or the network graph. In this network graph , each word from the text corpus can be shown as a node, and the edges between these nodes denote the strength between these words like how often they co-occur in the same sentence. To make the network graph visualization easier and intuitive, we built an interactive feature, which helped to highlight a specific node and its neighbours in the graph by selecting that specific node. To analyse the network graph in depth, we also looked at different centrality measures such as the betweenness centrality (BC) and eigenvector centrality (EC) of these words. Betweenness centrality shows how often a node (or word) acts as a bridge node, that is the number of times a node lies on the shortest path between other nodes. For explainability, this means that a node (or a word) with high betweenness centrality would have more control over the network. Another centrality measure that can be a good indicator of the influence of a node (or word) is eigenvector centrality. Therefore, a node with a high eigenvector centrality score must be connected to many other nodes who themselves have high scores. In the next section, we describe the visualizations generated by using these data analysis methods in the context of the first session.First, we did an exploratory visualization using this technical setup to see the frequently used keywords in different phases by different roles. As described in the Task Context Sub-section above, in the blue phase, the main objective was to discuss the steps in the learning activity. Each learning activity consisted of a sequence of interactions such as learner to teacher, learner with learning environment, material to learner and so on. The red phase as defined earlier was supposed to be a conversation about learning enhancing technology. The yellow phase as defined earlier was supposed to be a conversation on the interaction within the learning activity and aspects a teacher might want to know about them. For example, it can be engagement, social interaction or how students take initiative. To get a representative overview of these keywords, we examined the topical clusters obtained by using LDA and LSI in the red phase, which had a technological underpinning. We chose to go deeper into this phase because of our inclination towards technology. When we analysed the turn taking of different roles during the red phase, we found that the TEL LA advisor and the teacher had the most exchange of turns between them. Therefore, to further explore their roles and the usage of the words, we analysed the bigrams (two consecutive word combinations) of the words. We wanted to see the relationship between these words and phrases used by the two speakers  acts as a bridge node (or a node that has more control over the network). Another centrality measure that can be a good indicator of the influence of a node (or word) is eigenvector centrality. Therefore, a node with a high eigenvector centrality score must be connected to many other nodes who themselves have high scores. If EC is seen in Thus, in this study, we took a computational approach for prototyping our technical setup and advancing towards automatic collaboration analytics.RQ 1: “What co-located collaboration indicators have been detected from group speech data in past studies?”, we found that most studies are on non-verbal audio indicators . Most previous studies on co-located collaboration (CC) focused on spectral and temporal audio indicators [First, with the help of dicators ,20. Thesdicators ,24) on vRQ 2: “What collaboration analytics can be employed to analyse group speech data from co-located collaboration automatically?”. For this, we built a technical setup and conducted a field trial where we recorded sessions of a board game where people had to collaboratively design learning activities and each player was assigned a specific role to play beforehand. With the help of this collaborative task, first, we had an exploratory understanding of the collected audio data set. Then, we visualized the relationship between these words, apart from the representative topic modelling (which are certain representative word clusters) done in the past [To address this, we answered the past . First, the past . To undethe past ), which “What is the way forward once we have this technical setup ready and a tool ready to be used in different settings?” The first step for us is to use this across the other sessions for which we collected audio data and see if we can find some recurring themes. We also want to add further enhancements  and refine the visualizations by involving the two main stakeholders  for whom this was made. We want to understand the turn-taking patterns between roles further and how the conversation evolves around these turn-taking patterns. For now, it is a post hoc group collaboration analytics tool. To take it a step further, it would be interesting to add a module to detect the quality of collaboration. One possible step can be to compute the cosine similarity distance by comparing the text vectors of each role as computed by this technical setup to the expected contribution of the roles. This can give an estimate of how aligned the conversations are to the expected contribution for that role and an estimate of the collaboration quality if we can quantify the cosine similarity distance as a quality measure. Therefore, to proceed in this direction, we need to bootstrap the results of all these sessions to build models of different roles along with a human to form some expected standards (or words they would use) for each role during the CC task.Now, the next obvious question is: https://bit.ly/autocollabanalytics, last accessed on 30 April 2021)) to move towards automatic collaboration analytics right from audio data collection to generating meaningful visualizations.Therefore, for this article, our main contribution was twofold: identifying the gaps of the current co-located collaboration (CC) quality detection approaches from audio data and making a starting step towards an automatic holistic collaboration quality detection technical setup with the prototyping of our setup in the context of a CC task. This was a technical article stressing the different technological approaches  and verbal audio indicators . There have been many studies on the first category, but very few studies on the second category of analysing the content of the conversation. We found that with the maturity of automatic speech recognition systems, recently, analysis of the content of the conversations has picked up pace. Most studies analysing the content of the conversation looked at the high level topics and were lab based.Therefore, we took a step further to build a technical setup and conducted a field trial to analyse the richness of natural unstructured conversations and the strength of the links between these words and phrases used in the conversation context, and thus, we prototyped the tool to move towards automatic collaboration analytics. Here, we analysed the conversations during a board game while designing a learning activity where group members with different roles  interacted with each other. We found different interaction patterns between the teacher and the TEL LA advisor by analysing the word-level and phrase-level interaction during the technology related discussion phase of one collaboration session. Even though we were moving towards automated collaboration analytics, we found limitations in terms of automation with speaker diarization and data pre-processing.As mentioned in the Discussion, we want to enhance the technical setup further by understanding the conversations around these turn-taking patterns between different roles. The major outlook for the future will be to measure the quality of collaboration and give feedback. Due to COVID-19, we are also looking into adapting our approach to a remote (or online) setting from a face-to-face setting. Because of our modular approach, it will be easier to adapt everything in the technical setup except the speaker diarization. We will not need speaker diarization in an online setting, and it will be much easier to get different clean audio streams from each group member in an online setting.