Moreover, we discuss the major limitations in today’s IoT and highlight the required capabilities in the future. We illustrate this global vision with the help of two concrete instances of the hyper-connected IoT in smart cities and autonomous driving scenarios. Finally, we analyse the trends in the number of connected “things” and point out open issues and future challenges. The proposed hyper-connected IoT framework is meant to scale the benefits of IoT from local to global.All of the objects in the real world are envisioned to be connected and/or represented, through an infrastructure layer, in the virtual world of the Internet, becoming Things with status information. Services are then using the available data from this Internet-of-Things (IoT) for various social and economical benefits which explain its extreme broad usage in very heterogeneous fields. Domain administrations of diverse areas of application developed and deployed their own IoT systems and services following disparate standards and architecture approaches that created a fragmentation of things, infrastructures and services in vertical IoT silos. Coordination and cooperation among IoT systems are the keys to build “smarter” IoT services boosting the benefits magnitude. This article analyses the technical trends of the future IoT world based on the current limitations of the IoT systems and the capability requirements. We propose a  The Internet-of-Things (IoT) concept has been broadly adopted by heterogeneous communities due to its potential benefits. Progresses in technologies enabled the realization of today’s IoT services. Nevertheless, new technical capabilities are needed to realize “smarter” IoT services or even open possibilities not viable today.In the current IoT, multiple data sources contribute sensing information and the sensed data is gathered into a single cloud service. Let us consider the example use case of environmental monitoring for noise pollution measurement in a city. Data from sources such as noise sensors and people’s comments from social media are gathered in for the particular cloud service and these sources are used only for the city noise measurement purpose. In the future, on the other hand, produced data need to be shared among multiple applications. Therefore, the linkage between objects, devices, edge devices, actuators, agencies, and services needs to evolve to many-to-many instead of the current many-to-one or on-to-one linkages. This unique characteristic requires transparent discovery and information exchanges opening to more opportunities for information mash-ups among interdependent and symbiotic cloud services.hyper-connected IoT framework which connects in synergy multiple interdependent services. The first technical capability lacking in the present systems is the ability to orchestrate data analytics among distributed and federated IoT systems [This article points out the technical deficiencies in the current IoT and proposes a  systems , where t systems , real-tiAs extension of our previous work , this arThe remainder of this paper is formed by the presentation of the future IoT framework and its technical requirements  togetherhyper-connected IoT framework, proposed in our previous work [In this work, we present instantiations and technical capability advancements of the ous work , that enThe future IoT framework capabilities are horizontally placed on vertical IoT elements such as device, edge and cloud. Data is produced by sources through the observation of the real-world such as physical phenomena measurements. The capabilities are then fulfilled by functions that are executed either in the cloud or in the edge depending on the hardware resource requirements , real-time constraints, and/or optimization of network bandwidth . Moreoveservice-defined data analytics that configures data analytics as a topology of functions, each of which are placed on the cloud or edge (or even device) depending on the requirements of the service. For example, at the edge, some stream analytics are executed in order to reduce the amount of data or to apply privacy-preservation algorithm (such as data anonymization). The cloud, which is orders of magnitude richer in terms of physical resources, is then responsible for performing high-demanding computation for data analytics . The second capability is the information transparency that aims to make the data understandable and usable by heterogeneous IoT systems and applications by hiding the complexity of the IoT elements topology. The translation of raw data to a common form is made by the semantic mediation, usually left to the edge of the IoT , whereas the semantic interoperability is performed as cloud services. Finally, the third capability is the resource and context management that allows the handling of data not only as isolated pieces of data or isolated datasets but as a global virtual graph of things, having attributes representing the things’ status, linked with each other. The edge is left to associate the incoming real-world observations to a thing uniquely identified by a knowledge base. The data is then semantically discoverable at the cloud level, having the latter a broader view of the IoT elements, making possible a mash-up of heterogeneous information coming from heterogeneous data sources.A generic future IoT framework deployment is depicted in As an application scenario of the future hyper-connected IoT, we analyse an IoT framework for autonomous driving in large scale environments such as cities. We first describe a layered view of IoT for autonomous driving functionalities and then provide a federation view for scalability.things and external services layer and continues with network layer, IoT layer, and application layer.The things layer includes any possible actively or passively connected  things. The vehicles include autonomous cars, drones, public transportation vehicles, non-autonomous vehicles, and so on. External services can provide information related to traffic situations or other related information such as weather conditions. The network layer provides the communication capabilities to provide connectivity through advanced 5G technologies and ITS-G5 (Intelligent Transport Systems operating in the 5 GHz frequency band) for vehicle to vehicle or vehicle to road-side communication. The IoT layer has functional building blocks such as device management, context management, semantic interoperability, process and service management, security, and data analytics. Applications of autonomous driving systems are built on top of the IoT infrastructure and they may include services such as optimization of autonomous transportation using cars or taxis.Data analytics in the future IoT needs to adapt to the intrinsic distributed nature of IoT. Thus, the data is processed at every layer of IoT . This is not only for scalability reasons (such as minimizing network bandwidth by filtering or aggregate useless and redundant data) but also for taking into consideration the high dynamism of IoT infrastructure topology . Moreover, privacy requirements steer the decision about the allocation of processing tasks since it is always wiser to execute privacy preserving data processing as closest as possible to the generation of the data. The needed analytics are leveraged by IoT services with heterogeneous requirements such as geographical scope of data analytics  or Qualistream processing part which comprehends a filtering task that filters redundant or unnecessary data, and a privacy preserving task that anonymizes the observations (important for Wi-Fi monitoring). In the second category, continuing the example, we can include the batch processing part where historical data is aggregated and patterns are detected.As an example of a service-defined data analytics, we consider crowd mobility analytics  that estSemantic interoperability ensures consistency of data across interacting systems regardless of their individual data formats as these systems attribute the same meaning to the exchanged data. The semantics of data can be defined in a way that different data formats use a shared vocabulary and the vocabulary is specified in an ontology. The concept of semantic interoperability is applicable to all elements of the future IoT framework from data sources to cloud and applications.The capability of having transparent exchange is fundamental for the future hyper-connected IoT systems. This capability consists of semantic interoperability and semantic mediation. Interoperability is the ability of multiple systems to exchange data. Semantic mediation is the capability of transforming data coming from one system to make it useful to another system. To achieve interoperability cloud needs to have certain standards so that it can provide contextualized access to the applications. Conversion of data that has certain standards such as oneM2M [s oneM2M  to others oneM2M  can be ds oneM2M . ConsideAn example of a semantic approach to IoT interoperability is the combination of open standards such as OMA NGSI with the semantic concept of ontology see  such as ssn:Sensor is a class defined in the SSN ontology. For sake of readability, in this paragraph, we are omitting the namespace prefix, but they are all represented in Observation is the core element that bridges the physical phenomena sensed in the M3-lite taxonomy [QuantityKind . This taxonomy is fundamental to cover the many different sensors of various IoT deployments. Each Observation has a timestamped (Instant), a location (Point), a unit Unit, observed by a sensor (Sensor) that is the superclass for all kind of sensors as defined in the M3-lite taxonomy .In an ontology, rich and complex knowledge is represented as a graphs of nodes, expressing things, and links between nodes that express the relationship between two things. Things are categorized in classes and every thing is an instance of a class. A class may have sub-classes, and an instance of a class is also an instance of all the super-classes of such class. Classes and relationships are specified into taxonomies and an ontology can use classes and relationships from one or more taxonomies. Each class and property used into an ontology is uniquely identified by a namespace prefix and the class or property specific name. For example, taxonomy  via the EntityId and reports several ContextAttributes. EntityIds might have one or more DomainMetadata which specify metadata in common to all the attributes . In addition, each ContextAttributes might have multiple AttributeMetadata that are specific to such attribute.In the NGSI standard, each piece of NGSI context data refers to an identifier of the entity, namely isPattern (that is used to indicate whether EntityId identifies a range of entities) is set always to false. Each ssn:Observation is uniquely linked to a ContextAttribute that takes the qu:QuantityKind as Name, the class of dul:hasDataValue as Type and the ssn:ObservationValue as ContextValue. Furthermore, each ContextAttribute has one AttributeMetadata for the qu:Unit and another one for the time:Instant. The location of the sensor is stored as a DomainMetadata for all the attributes.The mapping of the FIESTA-IoT ontology with the NGSI format is shown in OMA NGSI is currently under study of the Context Information Management (CIM) working group of the European Telecommunications Standards Institute (ETSI) for evolving in a new version that incorporates linked data concepts is and formatted in NGSI-Linked Data (NGSI-LD) . Therefocontextualization process from data to services which requires the capabilities of resource-entity mapping and semantic-based discovery. In the traditional IoT framework, multiple entities which are objects in the real world  are considered together to provide resources  for a single purpose. For example, sensing readings from the temperature and humidity sensors in a smart home are only used to trigger the heating system. However, the future IoT platforms consist of multiple interdependent systems that collaborate with each other in a symbiotic manner to share all available resources. As shown in In the future IoT framework, the resource-context management refers to the The whole interconnection of resources creates a next generation of context, resembling a global distributed graph of information, that can be seamlessly accessed by any actors of the Internet-of-Things. Data requests are brokered by specific components, namely Next Generation Brokers (NG Brokers), that dispatch data requests to providers of the requested data and aggregate the responses before to return back to the requester achieving a transparent global IoT since the IoT topology is kept hidden from data producers and data consumers perspectives.knowledge extractions that combine sensed resources mapping them to common entity models and information model. This capability generates new metadata and attributes necessary for linking information into a context mash-ups. For instance, crowdsensed measurements of noise through smartphones [situation classification analytics components, of which output is either signalled to a human or automatically triggers actions. In addition to these specific contextualization computation, other stream analytics processing are executed for various purposes such as privacy preservation, data polishing, sensor fusion. As we have seen in analytics functions (or tasks) dynamically instantiated within the Artificial Intelligence (AI) layer federated among systems , might be physically distributed according to several optimization directives with different perspectives  orchestrSecurity layer, horizontally placed on all the components handling and exposing data, is managed by local administrations that allow the owner of the IoT deployments to have full control of the data, and consequently regulates analytics access and computation over the data.In this section, we are showing two examples of an actual realization of the hyper-connected IoT vision based on two real-world scenarios: a smart city and IoT augmented autonomous driving.As an example of future application that makes use of cross-cloud interoperation, we have implemented a smart city application, named Smart-City Magnifier. The application purpose is to analyse the situation of generic geographic areas and compute indicators representing the health of the urban and natural environments. The outcome of the indicators are then showed into a dashboard that highlights the critical situations such as emergencies , high crowd densities, and urban and rural pollution. The data is harvested from data sources belonging to different IoT domains hosted by separated IoT clouds and going through a process of associating observations to real-world things  and then analyse the situations related to those things.FogFlow [Data Contextualization that associates virtual entities to the incoming observations. Contextualizing, in this scope, is the act of inferring the real-world things  to which each geotagged observation belongs, making usage of the knowledge storage offered by external knowledge bases such as OpenStreetMap [Data Aggregation groups the incoming observations by the inferred things. The aggregation makes usage, for instance, of statistics means. The Indicators Computation is calculating smart city Key Performance Indicators (KPIs) [The interoperable data is then, on the one hand, handled on multiple IoT Broker instances federated in a topology of peers, and, on the other hand, orchestrated by the FogFlow  framewortreetMap . One obss (KPIs)  assessinThe output of the analytics is handled again by the federated IoT Brokers. The dashboard is then acquiring the information and visualizing it on a map presenting the alerts and warnings with the means of several visualization widget tools. The same visualization dashboard is used for visualizing all the situations inferred from the IoT silos integrated in Europe, Korea and New Zealand, with a total of 13 city areas . Since the analytics processing tasks are working on homogeneous semantically annotated data, the geographic scope of the dashboard map can simply be navigated to focus on a different city.crowd size estimation given by the service based on the level of Wi-Fi activities in the surrounding of the deployed Wi-Fi sniffers. Wi-Fi sniffers are statically deployed on the road-side as well as in the autonomous vehicles by including GPS sensor for knowing the positions of the vehicles.We built a Crowd Estimation and Mobility Analytics (CEMA) service which is beneficial for enhancing autonomous driving through IoT. Here, we focus on the This subsection describes the use of the proposed CEMA, federation, FogFlow, and SMG technologies in the smart mobility domain for IoT-enhanced autonomous driving. Integration of these technologies with other IoT and/or autonomous driving components makes the use of them possible in the pilot tests, which are conducted at the Technical University of Eindhoven campus in the Netherlands. The purpose is to estimate crowdedness in the university campus and leverage measurements coming from many vehicles to create a dynamic map of the crowd. By using the dynamic map, the autonomous vehicles can make routing decisions such as choosing a less crowded path or other decisions such as speed enforcement during certain events where many pedestrians exist.Instead of performing data analytics on the central IoT, an alternative approach is to migrate whole detection, processing, and analytics tasks toward the front-end devices, only the data analytics results are published to central IoT. In this case, lightweight edge-based crowd mobility analytics modules can be performed in a single device attached inside automated public transport. Similarly, an edge device is integrated with a GPS sensor and an inertial measurement unit (IMU) sensor for performing stop detection, passenger estimation, and passenger flow tracking. On-board processing and analytical approaches are more privacy-preserving and consume less energy for data communications. Experiments are conducted in the automated hanging trains system at Technische Universität Dortmund.We have performed experimental tests with the aim of analyzing the impact of a topology of federated IoT platforms instead of a single cloud IoT platform instance. In particular, we have made usage of the FIWARE foundation  and its x-axis the number of entities  requested in each query. The throughput is measured as the number of entity data (fixed to 20 attributes per entity) per second. Several sizes of the IoT system are tested: 100, 1000 and 10,000 entities. The requests are made using 100 parallel clients continuously requesting the systems. The plots show that the performances, both throughput and latency, are not affected much by a big volume of data exchanged, although the federated configuration requires more operations and generates more messages.We have then performed a second test for assessing the scalability advantages implicit of the federation. We have envisioned a scenario of 10,000 entities handled both by the centralized system and by a federation of 10 different providers, each one managing a disjoint set of 1000 entities. The test has been repeated for 10, 50 and 200 parallel clients , requesting 2, 8, 32, 128 and 512 entities per query. A federated platform has many advantages, for example: data providers can handle data locally preserving privacy and data sovereignty, a new data provider can join the federation (by performing an NGSI-9 RegisterContext) at any time since a discovery is performed for each request (we are not considering the case of caching), complex semantic discovery can be performed at discovery time, service-defined data analytics can be triggered on demand at the time of the query. In addition, a heterogeneous system can join the federation making usage of mediation gateways for translating the data format and interfaces methods.We have performed a trend analysis based on the reports from International Data Corporation (IDC) , Gartner(1)Data ownership management: For a future IoT where data is globally accessible and discoverable, special attention should be paid in order to assure that the producer of the data (or the owner of the observed things) keeps ownership of the data, especially for privacy-sensitive data. A study of the International Data Space Association (IDSA) [n (IDSA) ,37, whern (IDSA) , but, af(2)Privacy and security: With the realization of the presented capabilities, the future IoT will encounter new security and privacy threats. Every IoT layer, from application to devices, has peculiarities on the security risks and possible attacks. Considering the vertical elements in the bottom-up architecture, each level  has its own security requirements. Each level is exposed to various types of security threats and possible attacks. Currently, there is a lack of and a certain need for a dynamic IoT security model for enabling mission-critical applications  and expected advancements in the IoT systems. Furthermore, for building trust and secure relationships between the IoT components, proper identification and authentication capabilities, and cooperation among these techniques in the IoT platform are currently missing. On the other hand, preserving privacy of data in IoT is an open challenge. The existing privacy protection policies for today’s IoT include encryption, anonymization and obfuscation techniques, which are mainly for single services. However, new privacy preservation techniques in these interdependent services Critical real-time operation: The IoT of the future should be flexible and adaptable to sudden changes of the status and conditions of the infrastructures. This is due in order to have fast response to critical situations such as the increasing frequency of natural disasters due to the global climate change [e change . Infraste change  or easy-e change ,43 can h(4)Trustworthiness evaluation: A dual problem of the data access control is the control over data generation. Since the data is associated with real-world things reporting the status of them, only legit data sources should be allowed to report observations to a thing and, at the same time, the future IoT should be resistant to tampering attack. For that reason, it is a challenge to make a trustworthiness evaluation assessing which entity might be trusted and how trustful is the data generated [enerated .(5)Standardization: Different layers of IoT have been studied within many standardization activities. However, there is little consensus regarding which layers and relevant techniques should be standardized and which layers should remain open to be designed. In addition, governments showed their interest in standardization and their involvement implies innovation restrictions due to ever stricter regulations. New requirements for IoT are defined by IoT organizations such as OpenFog, the Industry 4.0, Made-in-China 2025 [ina 2025 , and theThese vast numbers of connected “things” and large data volumes are only possible with new technologies and standards. The unprecedented increase in “things” coming together with big data brings many new challenges and problems in connectivity, processing, memory, sensing, and actions that require enabling of the future IoT platform capabilities and 5G in order to lead the aforementioned expectations into a reality. Based on this observation, we point out some open issues in the future IoT as follows:A one-size-fit-all network management based on the same physical network infrastructure to exchange a variety of traffic and context is not efficient. Therefore, 3GPP considers a more flexible network slicing technology that is one of the important features in 5G. Currently, the 3GPP SA2 Working Group has defined the following three different categories of network slices based on the characteristics of service requirements: (1) extreme mobile broadband (xMBB), (2) ultra-reliable and low latency communications (uRLLC) or ultra-reliable MTC (machine-type communication), and (3) massive MTC (mMTC). First, high data rates and low latency are expected for the first type of services. Second, a critical level of latency and reliability are expected for the second types of services such as mission critical emergency response services. Third, machine-to-machine and device-to-device wireless communications are expected for the third types of service such as communications between lightweight IoT devices.The book by Vermesan and Friess  analysesTechnical capabilities exposed in this article are also addressed by specific studies. A work related to service-defined data analytics see  is propoFrom the application perspectives, the research efforts of Gerla et al.  and RahiIn this work, we have introduced a vision of the future Internet-of-Things which foresees a global interconnection and interworking of heterogeneous devices, systems and services. Several technical capabilities toward the future IoT laying on device, edge and cloud IoT layers are needed. Some of the key technical tipping points, such as service-defined data analytics, comprehensive of elastic edge-cloud orchestration, information transparency, data contextualization, and semantic discovery, are extensively analysed with the help of two concrete examples of hyper-connected IoT frameworks in the scenarios of smart cities and autonomous driving. The experiment results regarding the federation of IoT platforms show that the overhead for enabling a hyper-connection of systems is much lower than a big amount of data exchanges. In addition, the results demonstrate that the federation allows the scalability of IoT systems when the size of IoT scenarios increases. We have outlined challenges and open issues to be addressed by researchers and industry in the near future.As the next step for realizing the vision of the hyper-connected IoT, we are adopting the new standard of the ETSI CIM , which n


Wi-Fi-based indoor position sensing solutions have the advantages of easy integration in mobile phones and low cost by using existing Wi-Fi access points. The mainstream methods are commonly based on the received signal strength indicator (RSSI), which suffers from multipath interference in complicated indoor environments. Through the in-depth analysis of the multipath interference, an RSSI-assisted time difference of arrival (TDoA) method is proposed for Wi-Fi-based indoor position sensing in this work. The key idea is to compensate for the multipath interference in the received signals based on the coarse estimation using RSSI and TDoA calculation. A prototype system has been implemented to validate the proposed method. Experimental results have demonstrated the effectiveness of the proposed method, especially for handling the multipath interference with small propagation delay difference. Experimental results show that the indoor position sensing system can achieve a 90th percentile error of 0.3 m. The proposed method can also achieve moderate computational complexity and moderate real-time performance compared to other methods. In recent years, with the rapid development of personal mobile devices, the location based service has become increasingly indispensable in the daily life. The global position system (GPS)  has beenThe demands of indoor position sensing arise in various applications, such as human guiding , public The fundamental principle of Wi-Fi-based indoor position sensing is conducted in three phases. Firstly, Wi-Fi frames between the targeted mobile device and the APs  are collected as the raw data. Secondly, the key feature is extracted from the raw data which contains the position information. Finally, the feature information is processed using the position calculation algorithms to estimate the position of the targeted mobile device. Unlike the line-of-sight (LoS) signal transmission path in a GPS, indoor position sensing is confronted with complicated communication channel situations. The key challenges include multipath propagation, fading, noise and the environmental dynamics .A number of Wi-Fi-based indoor position sensing methods have been reported in the literature. Roughly, those methods can be classified into two categories, namely, the received signal feature-based methods, and the propagation time-based methods ,36,37,38The time of arrival (ToA) and the time difference of arrival (TDoA) methods are the commonly used propagation time-based methods ,36,37,38The mechanism of the position sensing accuracy loss due to the multipath interference effect is analyzed theoretically, and the multipath strength indicator is defined to measure the interference quantitatively.A novel RSSI-assisted TDoA method is proposed to mitigate the impact of the multipath interference. Especially, the proposed method is capable of handling the circumstances with small propagation delay difference.The prototype of an RSSI-assisted TDoA position sensing (RTPS) system has been implemented in a software defined radio (SDR) platform. The prototype system shows advantages of high accuracy, high robustness, and low computational complexity compared to other methods in the literature.In this work, a Wi-Fi-based indoor position sensing method with multipath interference mitigation is proposed. The major accomplishments of this work include:This work gives a new method to fuse the classical RSSI and TDoA methods for the performance improvement, with the cost of extra signal processing steps compared to the classical methods. Unlike the TDoA methods in the literature, the proposed method can be used to handle the multipath effect with very small propagation delay difference between the LoS and non-LoS signals. The key contribution of this work is to use the RSSI result to coarsely estimate the multipath interference, and then to use the coarse estimation information to improve the TDoA measurement performance. To the best of our knowledge, there is no similar method reported by other research groups that combines both methods in such a way.The rest of this paper is organized as follows. The proposed indoor position sensing system is based on the IEEE 802.11b  Wi-Fi prThere are a few factors that affect the TDoA estimation accuracy. This work focuses on the non-ideality caused by the multipath interference. It can be shown that the accuracy of the TDoA calculation is greatly affected by the multipath interference. The proposed RSSI-assisted TDoA method is introduced to mitigate this problem. In this section, the conventional TDoA method will be introduced first, and then the proposed RSSI-assisted TDoA method will be described in details.Although the target of this work is to provide a 2D position sensing method, the basic idea will be explained for the 1D position sensing case in which the mobile node moves along the line connecting two base stations. The presented method for a 1D case can be easily expanded to the 2D case.N is the number of non-LoS channels, To focus on the multipath effect, the non-ideal factors such as the frequency offset and the noise interference are not taken into consideration in this analysis. According to the IEEE 802.11b protocol, at the transmitter of the mobile node, the direct sequence spread spectrum (DSSS) PHY with the Barker code c, L is fixed. For example, when For simplicity, it is assumed that At the base station sides, the baseband signals received by the two receivers are described asFirst consider the situation that there is no multipath effect, and the two received signals are simplified toN elements from The received signals are quantized to two sequences The processing flow of TDoA calculation is shown in g method  is used.The time shift of the peak point on the fitted parabola is given by The distance to the base station is calculated asN is the length of correlation sequence, W is the noise bandwidth of white noise, and SNR is received signal-noise ratio. To find an appropriate N, the numerical simulation has been performed. N. In this simulation, the actual TDoA is 1 ns, and the SNR is set to 15 dB. Based on N is chosen to be 2200.As proved in , if therNow consider the situation with a multipath effect. There may exist multiple non-LoS channels in a certain environment. Note that the non-LoS signals fade quickly due to the reflection/refraction loss and the longer propagation paths compared to the LoS path. For a given short time interval in which the RSSI and TDoA are measured, usually only one non-LoS path needs to be considered, which may give the signal strength comparable to that of the LoS path. Without loss of generality, in the following analysis only the major non-LoS component is included in the received signal Thus, Equation  is simplRewrite Equation  as(12)yNow define the multipath strength indicator (MPSI) as follows, which can be used to measure the impact of multipath interference.Since the non-LoS path propagation distance is larger than that of the LoS path, it is expected After normalization, Equation  can be wWith the multipath interference present, the calculated peak cross-correlation and its neighbors, i.e., Obviously, the additional terms added to The above phenomenon suggests that if the negative non-LoS component in d between the mobile node and the base station is then obtained using the log-normal shadowing model [d from the transmitter, n is the path loss exponent. n are fixed for a system with given transmitted signal power, antennas and In this work, the received signal strength indicator (RSSI) is used to recognize the polarity of the ng model  describeL between the two base stations, when the mobile node moves between the base stations. If If If If No compensation will be applied for all the other conditions.For the system with two base stations, the estimated distances between the mobile node and the two base stations based on the RSSI are denoted as The values of In summary, the proposed RSSI-assisted TDoA method for the 1D position sensing is composed of the following steps, which is also shown in Step I: Use the cross-correlation calculation and the quadratic fitting to coarsely estimate the mobile node position Step II: Use the RSSI to coarsely estimate the mobile node position Step III: Based on the coarse estimation in two preceding steps, determine the polarity of the multipath interference in the received signals, and compensate for the multipath interference in the signals Step IV: Repeat Step I by using the signals after compensation The same method is used for the 2D position sensing, and the detailed processing procedure will be explained in To validate the proposed RSSI-assisted TDoA method, a prototype system has been implemented. In this section, the hardware and signal processing implementation of the proposed TDoA position sensing method are presented.The overall architecture of the proposed indoor position sensing system is given in The mobile node is implemented using an NI USRP-2920 device, and the related functional blocks are shown in r to generate the baseband signal y for the subsequent protocol parsing and the proposed RSSI-assisted TDoA method. The architecture of the baseband receiver in the base station is depicted in The base stations are implemented using NI USRP-2952R devices. Each USRP-2952R device is composed of two receiving channels, and each channel serves as a base station. Therefore, two USRP-2952R devices are needed to implement three base stations. Note that all the base stations need to be synchronized. The two receiving channels within each single device is inherently strictly synchronized. Two USRP-2952R devices are synchronized by deploying the master–slave mode. The slave USRP-2952R device is synchronized to the master device by receiving a time reference signal from the master through a coaxial cable. Three base stations are identical, consisting of the RF front end, the ADCs and the baseband receiver. The baseband receiver processes the received signal z of two sampling points spaced by N is expressed as followsThe phase recovery block is specially designed to compensate for the carrier frequency offset between the mobile node transmitter and the base station receivers. The frequency offset results in an additional envelope in the received signal, and will finally cause the drift in the cross-correlation calculation. The classic frequency offset correction algorithms are based on the Fast Fourier Transform (FFT), which requires a large number of data points. In this work, the Coordinate Rotation Digital Computer (CORDIC) algorithm proposed in  is adopt9rn=sn·exz. In the actual implementation, The envelope sampling block ensures that The 1D position sensing is quite simple, and can be performed directly following the procedure described in The conventional position calculating algorithms based on the TDoA include the Fang method , Chan meThe presented method can be easily extended to the 3D positioning solution by modifying the processing flow in The proposed RSSI-assisted TDoA method has been validated through both the numerical simulation and the experiments using the prototype system.t = −1 ns. The propagation channel between the mobile node and either base station consists of the LoS path and one non-LoS path with In order to evaluate the algorithm’s performance, the numerical simulation is performed using MATLAB. The simulation of 1D position sensing with one mobile node and two base stations is described in this section. In this simulation, the mobile node is placed at a fixed position, and the theoretical TDoA is set to To validate the proposed method, experiments on the prototype system are conducted in a meeting room. The 1D position sensing experiment setup with one mobile node and two base stations is shown in The RSSI values of two base stations are shown in The experimental setup for 2D position sensing is shown in The performance of the presented RSSI-assisted TDoA method is summarized and compared to three representative works in the literature, as shown in In this work, we analyzed how multipath interference affects indoor position sensing accuracy, and the multipath strength indicator is introduced to quantify the impact. An improved cross-correlation-based TDoA method is proposed by using RSSI calculation for coarse estimation. The main idea is to compensate for the multipath interference in the received signals with the RSSI values and the coarse TDoA results. To validate the effectiveness of the proposed method, a prototype system has been implemented in a software defined radio platform. Both the simulation and experimental results have demonstrated that the proposed method is effective to mitigate the multipath interference effect in position sensing with affordable computation overhead. Although the proposed method is validated based on the IEEE 802.11b Wi-Fi protocol, it can be easily transplanted to other wireless systems without any constraint on the signal type, channel bandwidth, MAC protocol, etc., which is valuable in IoT applications with self-organizing sensor networks.Future work will probably focus on the verification of the proposed method in real application scenarios, by building an indoor position sensing system based on real mobile devices and off-the-shelf Wi-Fi access points. The proposed method will be extended to 3D position sensing situations with more base stations deployed in a 3D indoor space. Also, efforts will be made to verify the method on other wireless communication protocols.


Speech recognition (SR) has been improved significantly by artificial neural networks (ANNs), but ANNs have the drawbacks of biologically implausibility and excessive power consumption because of the nonlocal transfer of real-valued errors and weights. While spiking neural networks (SNNs) have the potential to solve these drawbacks of ANNs due to their efficient spike communication and their natural way to utilize kinds of synaptic plasticity rules found in brain for weight modification. However, existing SNN models for SR either had bad performance, or were trained in biologically implausible ways. In this paper, we present a biologically inspired convolutional SNN model for SR. The network adopts the time-to-first-spike coding scheme for fast and efficient information processing. A biological learning rule, spike-timing-dependent plasticity (STDP), is used to adjust the synaptic weights of convolutional neurons to form receptive fields in an unsupervised way. In the convolutional structure, the strategy of local weight sharing is introduced and could lead to better feature extraction of speech signals than global weight sharing. We first evaluated the SNN model with a linear support vector machine (SVM) on the TIDIGITS dataset and it got the performance of 97.5%, comparable to the best results of ANNs. Deep analysis on network outputs showed that, not only are the output data more linearly separable, but they also have fewer dimensions and become sparse. To further confirm the validity of our model, we trained it on a more difficult recognition task based on the TIMIT dataset, and it got a high performance of 93.8%. Moreover, a linear spike-based classifier—tempotron—can also achieve high accuracies very close to that of SVM on both the two tasks. These demonstrate that an STDP-based convolutional SNN model equipped with local weight sharing and temporal coding is capable of solving the SR task accurately and efficiently. Automatic speech recognition is the ability for a machine to recognize and translate spoken language into text. It is a challenging task since the speech signal is high variable due to different speaker characteristics, varying speaking speed, and background noise. In recent years, artificial neural networks (ANNs), especially deep neural networks, have outperformed traditional Gaussian mixture models and became the predominant method in speech recognition area .ANNs are inspired by features found in brain. They consist of multiple layers of artificial neurons which are able to learn data representations from the input data by gradient descent algorithms , 3. In sFor these reasons, there has been a growing interest in spiking neural networks (SNNs) recently. Like in the brain, a neuron in the SNNs fires only when its membrane potential reaches its threshold. When a neuron fires, its post-synaptic neurons receive the spike and update their potentials. When implemented on neuromorphic platforms like TrueNorth , the SNNFor the speech recognition task, several SNN models have been proposed, which have either recurrent connections or feedforward connections. For the recurrent SNN models, a popular approach is the liquid state machine (LSM) –25, whicTherefore, for the purpose of both biological plausibility and energy efficiency, here we proposed a feedforward SNN with STDP and fast temporal encoding scheme for the speech recognition task. Our model was inspired by  and 31]31], it cOur network model consists of three layers, which is illustrated in the architecture diagram For the SNN to process, analog speech signal is needed to be encoded into discrete spikes. First we extract features from the raw signal to get a better representation, which is the first stage of almost any speech recognition system.As the encoder in human auditory system, the cochlea receives sound in the form of vibrations. When the sound vibration is transmitted to the basilar membrane, different points along the membrane resonate to specific frequencies. As a result, the hair cells protruded from the basilar membrane also have a tonotopic sensitivity, which can be modeled by an array of band-pass filters known as filter banks. The filter-bank-based Mel-Frequency Cepstral Coefficients (MFCCs)  is the mM × N array, where each row is a time frame and each column a frequency band. The dynamics of neuron  depends on the time frame m and the frequency band n.The sensitivity of a single auditory neuron depends not only on frequency, but also on time, as described by the spectro-temporal receptive field (STRF) , 39, whim, n) converts the intensity of the frequency band n in the time frame m into the time of the first spike. The higher the intensity, the earlier the neuron fires. With this temporal encoding scheme, our SNN is capable of high-speed and low-energy data processing compared to rate coding, since a post-synaptic neuron can encode the whole information from a pre-synaptic neuron as soon as the former receives the first spike from the latter, while the rate encoding can not estimate the information in a very short interval due to the non-instantaneity of mean firing rate.For each neuron in the input layer, they must convert the analog input signal into discrete spikes. Various neural encoding schemes can be used for this purpose. The most popular method is to convert the input into a Poisson-distributed spike train, which encodes the information in the neuron firing rates, while all the information possibly contained in the temporal structure of the spike train is neglected , 40. ThiVthresh, the neuron fires a new spike, and the membrane potential is reset to its rest level Vrest. The update of the membrane potential of an IF neuron can be described as [V is a vector of all neurons’ membrane potential, W is the input synaptic weight matrix, and S is a vector represents the spikes in the last time step. After each sample is processed, we reset the membrane potential V to Vrest, Vrest = 0.The spiking neuron model used in the convolutional layer is the integrate-and-fire (IF) model, which is simple yet captures some fundamental characteristics of real neurons. The IF model assumes that a neuron integrates input spikes from presynaptic neurons and increase its membrane potential each time a spike arrives. When the membrane potential exceeds a specific threshold ribed as , 31:V as a soft bound limits the weight value between 0 and 1. The learning process will be ceased when the change of weight values is so small enough (|Δw| < 0.01) that they have no effect on the final network performance on the test dataset.The weights of convolutional layer are initialized by drawing from a Gaussian distribution. When the network is in the training process, the weights are updated by the STDP rule , 8. HereTo make the feature maps more distinct and their responses sparser, we further use a mechanism of lateral inhibition, which plays an important role in the auditory cortex . With thThe pooling layer performs a pooling operation on the convolutional layer to reduce the dimension of the representation, see Each neuron in the pooling layer integrates inputs from one section in the corresponding feature map in the convolutional layer. The pooling neurons are not allowed to fire and their final membrane potentials are used as training data for a linear classifier. It should be noted that the pooling layer is not trained, and the value of its weights are fixed to 1. So the final potential of a pooling neuron can be seen as the spike number in its corresponding section and feature map in the convolutional layer. After the processing of each sample, the membrane potentials of pooling neurons are also reset.We evaluate the model with a linear classifier. In detail, the evaluation of the model includes three stages, see Our model was evaluated on the task of speaker-independent recognition of isolated spoken words with the TIDIGITS dataset  and the a+ = 0.004 and a− = 0.003.We used utterances of female and male adults from the TIDIGITS dataset, which includes more than 4000 samples from 200 speakers. The dataset was randomly ordered and split into training set and test set with the ratio 7:3. In the experiment, there were 40 neurons in the input layer. The convolutional layer consisted of 50 feature maps, and the IF neurons in which had thresholds of 23. The convolutional window size was 6 × 40, which made it span all frequency bands, and its stride was 1. We divided the convolutional layer into 9 non-overlapping sections to share weights locally, and each section had a length of 4. The weights of convolutional layer were initialized with random values sampled from a Gaussian distribution with mean of 0.8 and standard deviation of 0.05. The learning rates of the STDP rule were After the SNN was trained by STDP, the output of pooling layer was classified using a linear SVM. With the parameters listed above, the model was able to achieve a classification accuracy of 97.5% on the test dataset. For a more detailed analysis, We evaluated the performance of the model at different time points through the training process, as shown in To show the advantage of the local weight sharing over the global weight sharing, we compared the performances of the SNN model with both strategies. With global weight sharing, all convolutional neurons in a feature map share their input synaptic weights, which can be viewed as that the convolutional layer only has one section. While in the case of local weight sharing, the convolutional layer has multiple sections which all have their own set of shared weights. The performance comparison between SNNs with these two mechanisms is shown in To understand the forming process of receptive fields of the convolutional neurons, and provide insight into the dynamic learning behavior, we visualized the weights of the convolutional layer during the learning process. For this purpose, the weights of the convolutional layer were arranged into the convolutional window shape 6 × 40, and the weight values were used as the values of the corresponding pixels. Three randomly selected feature maps in the fifth section were visualized as examples in In the experiment, the classification on the MFSC features with SVM achieved an accuracy of 95%, which is worse than the classification on the output of our SNN. This difference of results can be intuitively observed with the help of t-SNE technique , which iAlthough the increase of data separability is also a feature of reservoir computing, they usually achieve this target by increasing the dimension of data. In contrast, our SNN maps the data into a lower dimensional space. With 9 sections in the convolutional layer and 50 feature maps, the dimension of the SNN output is 450, while the MFSC features have a dimension of 1640, so our SNN reduces the data dimension by more than 70%. The reduction of dimension helps to remove redundant information and reduces the required computing time and storage space. For example, by running on a workstation with the Intel Core i7-6900K processor (3.2 GHz), the training of SVM takes 70 seconds on the MFSC features, but only takes 2 seconds on the output of our SNN.To show the capability of the proposed method on a more difficult SR task, we also tested the model on the TIMIT dataset, which contains 630 speakers of eight major dialects of American English. From this dataset we selected 17 words shown in  which arTo demonstrate the SVM classifier is not the necessary for our spiking neural network model, we also test the more biologically plausible classifiers, here we used a popular spike-based linear classifier—tempotron—as an example. Tempotron is a biologically plausible supervised learning model, which can discriminate different classes of inputs based on their spatiotemporal structures rather than their mean firing rates . In the Spiking neural networks had been gradually drawing attention due to its potential of solving ANNs’ problems of biological implausibility and computational intensity. However, it is not easy to train a SNN well for typical pattern recognition tasks, and various training methods have been proposed previously . Many stThere were other studies which also use STDP as the learning rule –31, 40. Our model and the studies mentioned above have proven the effectiveness of STDP learning rule, but why STDP can work so well, since the construction of STDP-trained SNN models is often more heuristic than analytic? Actually, the learning process with STDP and lateral inhibition can roughly be seen as the sequential k-means (online k-means) algorithm . In the One of the most important features of our SNN is the time-to-first-spike coding, which is faster and more efficient than the traditional rate coding. However, this coding scheme is ideal and highly simplified, since it assumes that all information about the stimulus is contained in the time of the first spike of a neuron, and all the following spikes are neglected. Rolls et al.  comparedWith regard to the application in industry, our model has the potential to be implemented on neuromorphic chips like TrueNorth  or LoihiPrevious experimental studies had shown that there are both feedforward and feedback circuits in the auditory pathway , while oTo provide an alternative speech recognition solution to ANNs which is biologically implausible and energy-intense, we proposed an STDP-based SNN model with the time-to-first-spike coding scheme and the local weight sharing strategy. It can achieve high accuracies on two speech recognition tasks. By adopting STDP learning rule and the temporal coding scheme, our SNN is able to learn acoustic features fast and efficiently, and make the speech data low-dimensional, sparse, and more linearly separable. Compared to global weight sharing, the proposed local weight sharing is more suitable for learning the features of speech signals. Moreover, our model can achieve comparable performance to traditional ANN approaches when using SVM as a classifier, and can also work well when using the spike-based classifier—tempotron. Therefore, in practice, due to the spike-based computation, our model with tempotron can be implemented on neuromorphic chips easily as a speech recognition solution with ultra-low power consumption. In summary, our study shows that a biologically plausible SNN model equipped with STDP, local weight sharing, and temporal coding has the ability of solving speech recognition tasks efficiently.


The demand for mobile data communication has been increasing owing to the diversification of its purposes and the increase in the number of mobile devices accessing mobile networks. Users are experiencing a degradation in communication quality due to mobile network congestion. Therefore, improving the bandwidth utilization efficiency of cellular infrastructure is crucial. We previously proposed a mobile data offloading protocol (MDOP) for improving the bandwidth utilization efficiency. Although this method balances a load of evolved node B by taking into consideration the content delay tolerance, accurately balancing the load is challenging. In this paper, we apply deep reinforcement learning to MDOP to solve the temporal locality of a traffic. Moreover, we examine and evaluate the concrete processing while considering a delay tolerance. A comparison of the proposed method and bandwidth utilization efficiency of MDOP showed that the proposed method reduced the network traffic in excess of the control target value by 35% as compared with the MDOP. Furthermore, the proposed method improved the data transmission ratio by the delay tolerance range. Consequently, the proposed method improved the bandwidth utilization efficiency by learning how to provide the bandwidth to the user equipment when MDOP cannot be used to appropriately balance a load. In recent years, the types of mobile data have diversified due to improvements in mobile device performance . FurtherTo balance the load, a method is used that automatically sets an optimum network according to a dynamic change in mobile data traffic . In anotIn addition to the aforementioned method, there exists a transmission-rate control model focused on content . In 10]10], the On the other hand, reinforcement learning has been applied in various fields in recent years . ReinforIn this paper, we propose and evaluate a delay-tolerance-based mobile data offloading method using deep reinforcement learning for improving the bandwidth utilization efficiency of cellular infrastructure and solving the traffic demand locality. If an agent can learn how to balance the load properly, considering various situations such as traffic model of the UE, topology structure, and eNB loads, proper balancing can be achieved under conditions for which time-wise offloading of MDOP cannot balance the load properly. Our method performs transmission rate control by determining the priority for allocating the bandwidth to the UE as per the quality of service level (QoS level) based on the content delay tolerance of the UE and the conditions of the eNB such as the load.In the proposed method, we apply deep reinforcement learning to load balancing in order to improve the bandwidth utilization efficiency by enabling appropriate transmission rate control even in situations where appropriate load balancing control is difficult in time-wise offloading of MDOP. We expect that improve the efficiency of accommodating mobile data to the communication infrastructure by balancing the eNB load. The time-wise offloading of MDOP offloads the traffic by allocating bandwidth with a focus on delay tolerance of content and improves the bandwidth utilization efficiency. On the other hand, there is a problem that it is difficult to control appropriately with rule-based control in the diversified network environment. To get an approximate solution of appropriate control and improve the bandwidth utilization efficiency in the various network environments, we use deep reinforcement learning for mobile data offloading. Our method applies to the time-wise offloading of MDOP as a transmission rate control model that takes into consideration the delay tolerance. The details of the MDOP are described in A learning flow of our method is presented in The RL server determines the priority of the bandwidth allocated to the UE according to the environment information. We define this priority as the QoS level. Subsequently, the MDOP server allocates the bandwidth to the UE based on the determined QoS level. The UE then sends the data according to the allocated bandwidth based on the QoS level. Finally, the RL server learns the priority for allocating the bandwidth from the eNB’s load fluctuation caused by the sending data of the UE and constructs a transmission rate control model.Previously, we proposed an MDOP that balances the load of the eNB while taking into consideration the locality of the demand ,6. FigurIn MDOP, an MDOP server periodically collects content information from UEs, load information from eNB, and performs band allocation. When an MDOP server notifies the UE of the allocated band, the UE can transmit content with the allocated band. When band allocation is performed, the maximum band is allocated to UE if the content does not have delay tolerance. Also, if the content has delay tolerance, the bandwidth is divided equally. MDOP controls the transmission rate to accommodate a load within the control target value. We defined an ideal load as the control target value. The ideal load is set in order to prevent the occurrence of situations where packet loss and the allowable amount of eNB is exceeded when burst traffic occurs. MDOP reduces the locality of the demand by accommodating the eNB load within the ideal load to smooth the eNB load. When the content delay tolerance is exceeded, the content is transmitted at the maximum transmission rate without taking into consideration the ideal load.Similar to the problem of the previous MDOP, the delay tolerance of the content is not taken into consideration. Therefore, the short delay tolerance content has the possibility to exceed delay tolerance because the same control is applied to all the content regardless of the delay tolerance. The proposed method focuses on the delay tolerance of the content and clarifying the priority to assign the bandwidth. The proposed method can balance the time-wise concentrated load by realizing the delay-tolerance-based bandwidth allocated method of UE. By using deep reinforcement learning, it is possible to determine the appropriate priority of bandwidth allocation under various situations when deciding the priority.We determine the transmission control of the UE by using deep reinforcement learning, and the RL server learns using a double deep Q-network (DDQN) . DDQN is network . Althoug to DDQN . Duelingracontent), the content delay tolerance (dtcontent), and the destination eNB ID. Moreover, the eNB information is that gathered by the MDOP server, such as the available bandwidth and current load of the eNB. Defining the states, actions, and reward is essential for learning using deep reinforcement learning. As the input parameter of the RL server, the UE information consists of the control target UE and the UE connecting to the same eNB. Although the QoS level can be determined based only on We define the priority of the bandwidth allocated to the UE as action Algorithm 1 Reward function of the proposed method1Current load of connecting eNB: 2Ideal load that is control target value: 3Available bandwidth of connecting eNB: 4Select action at time 5Episode end time: 6Normalization variable: 7ifthen8 9else10ifthen 11  12else 13  14end if 15end if16returnFinally, the reward ϵ. The linear decay ε-greedy policy decreases ε linearly. We decreased ε linearly from 1 to 0.01.We performed two evaluations to confirm the performance of the proposed method. Firstly, we evaluated whether the state, action, and reward are appropriate as definitions for the RL server to achieve the goal, which is to eliminate the temporal locality, in lization  for obtaIn this evaluation, we used CPU i7-6700k for learning. Besides, we used Scenargie, which is a network simulator, to accurately reproduce the long-term evolution (LTE) environment . FurtherFirst, we performed a basic evaluation to confirm that the proposed method can perform appropriate control in the scenario where the priority to be allocated is explicit. We evaluated the cellular infrastructure utilization efficiency of the proposed method. In this evaluation, the proposed method, the timewise-offloading of MDOP, and no-control are compared. We created a scenario different from the basic evaluation for this evaluation. Secondly, we analyzed how the traffic exceeds Finally, we analyzed whether the proposed method allocates bandwidths to the UE while considering the delay tolerance. Based on the above results, we confirmed that the proposed method improves the bandwidth utilization efficiency of cellular infrastructure by allocating bandwidth while taking into consideration the delay tolerance. In reinforcement learning, the reward function is an important factor for evaluating the action. In this paper, we applied only suppression of eNB load within We proposed a delay-tolerance-based mobile data offloading method using deep reinforcement learning. Principally, we applied deep reinforcement learning to the time-wise offloading of the MDOP to balance the load by taking into consideration the delay tolerance of the content. In our method, the RL server decides the priority of the bandwidth to be allocated to the UE based on the content delay tolerance, the remaining amount of content, and the load of eNB. The MDOP server then performs bandwidth control and load balancing. Initially, we evaluated whether the proposed method can assign a QoS level by considering the delay tolerance in a basic evaluation. The obtained result demonstrated that the proposed method can learn how to allocate the QoS level for balancing the load. Moreover, we compared the cellular infrastructure bandwidth utilization efficiency of the proposed method, the time-wise offloading of MDOP, and no-control. As a result of the evaluation based on the assumption that there are four types of contents with different delay tolerances, the proposed method decreased the traffic in excess of the control target value by 35% as compared with the time-wise offloading of MDOP. Furthermore, on the basis of the maximum excess traffic amount and the maximum excess time, we confirmed that the proposed method further resolves the temporal locality. Moreover, the proposed method increased the ratio of data sent by the delay tolerance as compared with the time-wise offloading of MDOP. Therefore, we opine that the cellular bandwidth utilization efficiency has been improved by allocating the bandwidth while taking into consideration the delay tolerance and load balancing. Although we show the effectiveness of the proposed method about load balancing by focusing on delay tolerance, we need to consider the installation location of MDOP server and RL server and environment information collection method in order to apply the proposed method to a real environment. Furthermore, the current learning model focuses on UE present in one eNB, and it is not easy to use the current model in the situation where there are multiple eNBs to be considered in a real environment. In the future, we will evaluate and consider our method in multiple eNBs, reduce learning cost, and consider the proposed method’s design to apply to a real environment. Moreover, we intend to evaluate the mobility model of UE using vehicles and the traffic model, including various delay tolerances to demonstrate the effects of the proposed method in a real environment.


Environmental perception plays an essential role in autonomous driving tasks and demands robustness in cluttered dynamic environments such as complex urban scenarios. In this paper, a robust Multiple Object Detection and Tracking (MODT) algorithm for a non-stationary base is presented, using multiple 3D LiDARs for perception. The merged LiDAR data is treated with an efficient MODT framework, considering the limitations of the vehicle-embedded computing environment. The ground classification is obtained through a grid-based method while considering a non-planar ground. Furthermore, unlike prior works, 3D grid-based clustering technique is developed to detect objects under elevated structures. The centroid measurements obtained from the object detection are tracked using Interactive Multiple Model-Unscented Kalman Filter-Joint Probabilistic Data Association Filter (IMM-UKF-JPDAF). IMM captures different motion patterns, UKF handles the nonlinearities of motion models, and JPDAF associates the measurements in the presence of clutter. The proposed algorithm is implemented on two slightly dissimilar platforms, giving real-time performance on embedded computers. The performance evaluation metrics by MOT16 and ground truths provided by KITTI Datasets are used for evaluations and comparison with the state-of-the-art. The experimentation on platforms and comparisons with state-of-the-art techniques suggest that the proposed framework is a feasible solution for MODT tasks. Automated driving systems capable of performing all driving tasks under all roadway and environmental conditions manageable by a human counterpart, is classified as the highest level of automation by the Society of Automotive Engineers (SAE) International . AlthougThe prominent LiDAR-based MODT approaches follow tracking by detection methods. In this method, a set of trackable objects is identified by a detector. Subsequently, the detected set of objects is processed for pose and centroid estimation, accompanied by a known model fitting. This generates a valid measurement of an object which attributes further evolve over time. These measurements are fed to the state estimation filter to predict the kinematic states of the objects. The optimal Bayesian filters are utilized to estimate the possible evolution of object states in the presence of uncertainties. Later, a data association process is done to assign the detected object to either an existing track or initiation of a new track. To ensure that the unique identities of detected objects are maintained in the cluttered environment, Bayesian probabilistic approaches are deployed. Finally, a track management module is used to heuristically maintain the tracks and to cancel out the spurious ones. However, the most challenging part in LiDAR-based MODT frameworks is efficient object detection and classification. Furthermore, practical implementation of MODT requires real-time performance in cluttered urban environments, using limited computational resources.In this work, a complete MODT framework is proposed that relies on multiple LiDARs for perception. The detection component involves the slope-based ground removal of LiDAR point clouds, and a 3D grid-based clustering technique for segmentation and classification of objects. This enables the detection of trackable objects under elevated structures. Whereas the tracking component operates on Joint Probabilistic Data Association Filter (JPDAF) coupled with Interacting Multiple Model (IMM), and Unscented Kalman Filter (UKF). Furthermore, the inherent shortcomings of JPDAF pertaining to the combinatorial explosion problem  is also The remainder of the paper is structured as follows: in The objective of MODT is to determine the trajectories of detected objects and maintenance of their unique identity throughout the input sequence. The MODT methods are generally categorized into online and batch methods. Online methods rely on past frames up to the present frame information to solve tracking problems, whereas, batch methods utilize the entire sequence or the information from the future frames to assign the detections iteratively. Despite the fact batch methods are superior in accuracy, online methods are preferred for real-time applications, as tracking results are instantly available in the current frame . AlthougA feasible substitute to address camera-related limitations is to use LiDAR as a primary source. LiDAR provides a sparse spatial information of the environment that imposes less of a computational burden, making it viable for onboard computers. However, this solution comes with a tradeoff against the accuracy of object detection and classification. In addition, sparse and occluded information give rise to measurement uncertainty and a number of false detections. This requires additional treatment of detected objects for appearance-related attribute estimations. In MODT, data association of object detection in consecutive frames play an essential role in maintaining a correct trajectory and unique identity. To perform best associations, track motion models  and the The uncertainties pertaining to the measurements and state estimations are generally handled by Bayesian probabilistic approaches . The unThe proposed MODT architecture is implemented and tested on two platforms, a TUCSON and an IONIQ by Hyundai Motors , shown in The computational environment used for developing and evaluating the proposed MODT framework is constituted by a desktop computer having an Intel Core i7-7700 CPU with 16 GB of RAM, running Robot Operating System (ROS) “Kinetic Kame” middleware on top of Ubuntu 16.04.5 LTS. Moreover, Jetson AGX Xavier is used for the computation time evaluations, that runs ROS “Melodic Morenia” middleware on top of Ubuntu Linux 18.04.1. Furthermore, parallel processing or GPU support on both computing environments is avoided.The detection component of the proposed MODT scheme pre-processes the LiDAR point cloud into cloudlets that correspond to potential trackable objects. The pre-processing stage includes estimation of initial parameters and bounding box fitting. This information is fed forward to the tracker module for further processing. The detection component operates on a series of steps including, LiDAR calibration, ground extraction, point cloud clustering, and bounding box fitting, as shown in The platforms used in this work are equipped with three VLP-16 Velodyne LiDAR sensors. Each sensor generates point cloud with respective reference frame. The transformation matrices are estimated by the procedure expressed in , such thThe ground classification is referred to as the process of separating ground and non-ground point clouds. Methods to address this task are categorized as: (a) height threshold-based , (b) griThe task of ground classification from point clouds as a pre-processing for MODT systems has recently been approached by the abovementioned methods ,25,26,27The approach adopted in this work is a grid-based approach that considers a non-planar ground. The flow of the process followed in this module is shown in A similar implementation for ground classification is adopted in , howeverThe point cloud clustering step is the process of grouping and labeling the points possibly pertaining to a single object . The conThe centroid-based algorithms  include In this work, the clustering of point clouds is carried out through a hierarchy-based method. The flow of the clustering process is shown in An index cell is selected, and its 26 adjacent cells are searched for a minimum number of points. The qualifying neighboring cells are marked as the cluster members and are indexed for subsequent search. Labeling of cells are carried out to distinguish the clusters and to ensure that every cell is searched only once. The time complexity of clustering module is handled by setting a threshold limits x, y, and z dimensions. The x dimension is the heading of vehicle in the LiDAR sensor frame, the positive and negative limits are set based on the maximum distance at which identifiable cluster can be formed. Similarly, y dimension limits are set to a distance that minimizes the clustering of outliers like buildings and vegetation, considering a reasonable distance for crossroads. Moreover, limits in the z dimension are set to be little higher than the tallest trackable object. With these clustering limits in place, clustered point cloud is efficiently obtained.The point cloud segmented into clusters in a 3D grid is converted into clustered point clouds, where each cluster is associated to a potential trackable object. The dimension filter is used to filter out the outliers and to classify the objects. Furthermore, elevation and number of points of a cluster are also considered as parameters. First, clusters with dimensions exceeding the smallest and largest trackable objects’ dimensions are filtered out. Similarly, elevated clusters not satisfying the ground threshold limits are filtered. The clusters qualifying for trackable objects are proceeded to the bounding box fitting and classification stage.The box fitting and classification of objects in a LiDAR point cloud is a challenging task, mainly because of occlusions, that is any obstruction in the line of sight of LiDAR. Box fitting of point clouds not only helps in finding the objects’ centroids that can be efficiently tracked but also provides an initial pose estimation. This task is addressed by either (a) model-based, or (b) feature-based approaches. Model-based estimations tend to match the raw point cloud with a known geometric model . The posSince the objects in LiDAR point cloud measurements are always under occlusion, the minimum rectangle area alone cannot serve the purpose of bounding box fitting, and eventually affect the incorrect pose estimation. Hence, L-shape fitting is carried out over minimum area rectangle fitting to correct the pose of box , as showFirst, the dimensions of the clustered point cloud are calculated where length and width specify the initial area of the fitting rectangle in 2D. Then, the farthest two points of the clusters are found and a line between them is calculated. Finally, the most deviated point from this line is examined within the clustered cloud. This results in three points that correspond to the three corners of the actual pose of object. The rotation of the rectangle or pose of the object is estimated by calculating the angle between the initial edge of the rectangle and to the line joining the farthest point from the diagonal line to its either end. The height information is used as calculated from the dimensions of the point cloud to form a 3D bounding box. This procedure results in a box that represents an object in the LiDAR point cloud, and the location of object is indexed to the centroid of the box. At the end, the dimensions of the box are used to classify the object as vehicle or pedestrians for tracking initialization. For visualization, the bounding boxes matching strict dimension rules are replaced with 3D CAD models. In summary, considering the possibility of multiple LiDARs being used for measurements, the extracted clouds are calibrated and merged, such that all the measurements are referenced to the center of the ego-vehicle. Slope-based approach is adopted to segment the point cloud into ground and non-ground measurements. The point cloud pertaining to the non-ground point cloud is treated with a 3D connected component method for clustering. Subsequently, the filtered clusters of the point cloud that are potential trackable objects, are processed for box-fitting. A combination of minimum area rectangle and L-shape fitting process is applied to fit a 3D bounding box with an estimated pose of an object. The last step of the detection component is the assignment of classes based on the strict dimension criteria. The result of the detection component is a list of bounding boxes of known position, centroids, pose, and possibly class of the objects, that are fed as measurements to the tracking component for further processing.The tracking component of the proposed framework uses the measurements from the detection component to determine the location, path, heading angle, velocity, and angular velocity of the detected objects. Moreover, all the tracked objects essentially need to use a unique identity. As in the context of environmental perception, the said measurements are useful only if these can be incorporated in the ego-vehicle’s control and decision making. The task of tracking objects in the urban scenario using LiDAR need to address several challenges including, occlusion, dissimilar motion patterns of objects, and clutter. The term occlusion refers to the possibilities of incomplete spatial information either due to the pose of the ego-vehicle, or an object obstructing the measurement. This may result in partial or complete occlusion of an object. The partial occlusion can be handled by the detection component of the framework. Also, a complete obstruction of the tracked object contributes to the location uncertainty. In addition, the objects in the urban scenario tend to follow dissimilar motion patterns, like consistent variation in pose and speed. Moreover, many objects can exist close to each other forming clutter, contributing in the rise of uncertainty in the measurements.The tracking component needs to address the presence of two prominent uncertainties that are due to motion and clutter. Hence, two filters are set in place to address the both uncertainties while tracking. The filter for motion uncertainties incorporates a system with multiple motion models to track objects with uncertain motion patterns. Whereas, the clutter filter utilizes probabilistic data association approach to address uncertainties that arise when the tracked objects are closely located. Both mentioned filters however can be combined into a single tracking scheme with filtration performed at different steps. A similar tracking approach is adopted in ,13,34,40The implementation of IMM-UKF-JPDAF is an approach to efficiently address the problem of recursively estimating states and mode probabilities of targets, described by a jump Markov non-linear system, in the presence of clutter. Let the non-linear stochastic state space model be represented by Equations (1) and (2), forming a set of The elements essed in . A similessed in  as a fivIn this step, the initial state and covariance corresponding to each model filter is probabilistically mixed with the state and covariance of the previous time stamp:In Equations (4) and (5), The conditional mode probabilities depend on the priori mode probabilities, Once the initial states and covariances are obtained from the interaction step, prediction step is executed. Each model individually predicts the current state through respective stochastic non-linear model. Due to non-linearities, UKF is deployed to estimate the predicted states and covariances :Xk−1= for more than five consecutive time steps. The states of all tracks are traversed in every time step with threshold Euclidian distance limit of less than one meter. Among the qualifying set of tracks only the track with highest maturity count is retained and rest are pruned out.The state space models cannot accurately update the parameters of yaw, velocity, and angular velocity. For this matter, the track history is maintained, and previous five states’ information is utilized to estimate them. Moreover, the first five frames after initialization of the track ensures more exact parameters estimations of newly validated track. The heading information from the detector alone cannot be relied due to the phenomenon of occlusion. Hence, the heading of an object is estimated from both, the detector and the track history of the object, making it more reliable. Therefore, at every update step, the kinematic parameters estimated by incorporating the track history are integrated into the object states for more accurate prediction in the subsequent time step.In summary, an IMM-UKF-JPDAF is implemented as a combination of two filters to address the uncertainties pertaining to occlusions and clutter. The JPDAF shortcoming of combinatorial explosion is approached by mitigation strategy of clustering. The tracking component gets the object attributes as a structured data and through recursive operation of the proposed architecture estimates updated object states and tracks, covariances, and modes with respect to ego-vehicle reference frame. The object states are represented by position of object in a 2D space, heading of object, and linear and angular velocities. Whereas covariances and modes describe the degree of uncertainty and motion pattern  respectively. This data is managed and maintained through a track management module to produce an actionable information in a computationally efficient manner and can be shared across the platform over V2X.The method for evaluating this framework is of MOT16 as proposed by . HoweverTo evaluate the tracking framework against non-synthetic data, KITTI datasets  are usedIn this work, a criterion is proposed that overlaps the ground truth information pertaining to the camera and LiDAR frame as a reference. Hence a subset of ground truth is attained such that: (a) object is visible to both sensors (front of ego-vehicle), (b) existence of object within 30 m range from the sensor, and (c) the life time of a track is specified by the duration of first two conditions being true. The raw data provided by the KITTI datasets under the category of ‘City’, with the ground truth annotations are used for being more relevant to this implementation. However, the objects of type ‘Tram’, ‘Misc’, ‘truck’, and ‘person sitting’ are excluded for evaluation, and contribute to The KITTI Dataset provides ground truth for tracking information structured in XML format. Moreover, provides support to generate similar XML file directly from tracking algorithm. To perform the evaluation and benchmarking, the tracking component is programed to produce an XML file with tracking information similar in format to that of ground truth. In addition, MATLAB wrapper is also offered by KITTI Dataset to extract tracking information from the XML files to perform the metrics evaluations. The evaluation results of 10 dataset sequences are tabulated in The evaluation metrics in The datasets provided for evaluation purposes are generally based on single LiDAR sensor measurements, with limited vertical FOV. This restricts the intervention of elevated structures like traffic lights, bridges, and trees to be a part of LiDAR measurements within the effective horizontal range (20–40 m). However, measurements pertaining to these structures become significant when the sensor is tilted, or a combination of multiple sensors are used to capture most part of the surroundings. Furthermore, an assumption of height threshold for measurements contribute the similar problem to that of assuming a planer ground. That is, setting a vertical threshold limit for the height of measurement might work for a planer ground but suffer in a non-planer ground scenario. Here, an efficient clustering technique is required to detect and track an object that exists under the elevated structures. Addressing this issue is one on the main contributions of this work, as former works generally rely on vertical threshold limit. The proposed MODT method is tested on the platforms in the scenario where roadway is covered with trees on the top, while the trackable objects exist underneath. The limits for ground classification and clustering are set to 30 m from platform  and 10 m . The tracking evaluations for datasets are carried out on desktop computer; however, time complexity is also measured on the Jetson board. The time consumed by individual components of the algorithm while executing in respective computational environment is presented in The overall performance of the algorithm in terms of metrics pertaining to accuracy, precision, and quality are comparable to state-of-the-art MODT paradigms. However, the datasets provided by KITTI Datasets are underprovided with elevated structures and non-planar grounds to test the true potential of the algorithm. On the other hand, the factors affecting the performance are the result of compromise made against the time complexity. Despite the optimized code, the execution times in Jetson exceeds the sampling time of LiDAR in datasets with increased number of tracked objects. This however is addressed by the tracker module, as the rate of prediction is coupled with the delay in arriving measurements.LiDAR-based MODT approaches are relatively scarce in literature compared to camera- based ones. Furthermore, LiDAR-based schemes are rarely evaluated on MOT16 metrics, except for the work reported in ,9,10. BeThe proposed architecture possesses competitive performance metrics to the relevant former work presented in . The evaThe framework proposed in  is evaluThe work of  utilizesIn this work, a computationally efficient MODT algorithm for a multi-LiDAR setup is proposed, which can perform in real-time on a vehicle embedded computer. The slope-based ground classifier considers a non-planar ground assumption. Furthermore, the object detector relies on a flexible dimension filter to classify trackable objects, where the LiDAR point cloud is efficiently clustered using a 3D connected component method, with an assumption of elevated structure presence. Moreover, the proposed combinatorial explosion-aware IMM-UFK-JPDAF-based tracker operates in the presence of uncertainties to perform efficient tracking. In addition, the MOT16 metrics established for performance evaluation are used to demonstrate state-of-the-art results against non-synthetic datasets. The main objective of this work was achieved with an implementation of MODT algorithm on two slightly dissimilar platforms.In future, it is intended to further optimize the ground classification and tracking components and to replace the detector with more robust mechanism capable of performing object classification task. Moreover, it is envisioned to spare computational resources to incorporate visual object classification and utilize LiDAR-based spatial information for tracking.