Optical burst switching (OBS) networks have been attracting much consideration as a promising approach to build the next generation optical Internet. A solution for enhancing the Quality of Service (QoS) for high priority real time traffic over OBS with the fairness among the traffic types is absent in current OBS’ QoS schemes. In this paper we present a novel Real Time Quality of Service with Fairness Ratio (RT-QoSFR) scheme that can adapt the burst assembly parameters according to the traffic QoS needs in order to enhance the real time traffic QoS requirements and to ensure the fairness for other traffic. The results show that RT-QoSFR scheme is able to fulfill the real time traffic requirements  ensuring the fairness for other traffics under various conditions such as the type of real time traffic and traffic load. RT-QoSFR can guarantee that the delay of the real time traffic packets does not exceed the maximum packets transfer delay value. Furthermore, it can reduce the real time traffic packets loss, at the same time guarantee the fairness for non real time traffic packets by determining the ratio of real time traffic inside the burst to be 50–60%, 30–40%, and 10–20% for high, normal, and low traffic loads respectively. Optical Burst Switching (OBS)  network Accordingly, all these features indicate that the OBS network can be fundamental infrastructure of next generation optical Internet. Stability in the Internet network performance is a significant issue. It depends on the Quality of Service (QoS) that should be guaranteed to support thigh priority traffic categories, such as Constant Bit Rate (CBR), Variable Bit Rate (VBR), and depends on fairness among other traffic types that should be also ensured. In this paper we propose a novel Real Time Quality of Service with Fairness Ratio (RT-QoSFR) scheme to adapt the burst assembly parameters according to the real time traffic QoS needs, and simultaneously to ensure the fairness among other traffic types for enhancing the real time traffic QoS requirements over OBS network.RT-QoSFR can guarantee the entire delay of OBS network such that it does not exceed the MaxCTD parameter value in the real time traffic. Furthermore, it can reduce the real time traffic packets loss and guarantee the fairness for non real time traffic packets. Moreover, RT-QoSFR guarantees the stability in the performance of the network, the delay requirements, and ensures the fairness between real time traffic and non real time traffic, which provides a better QoS.The proposed schemes have been studied using the simulation model with two types of traffic , four values of MaxCTD, two value of burst size, and two different topologies . The objective of these scenarios is to demonstrate the possibility of the proposed scheme to work under various conditions.The remainder of the paper is structured as follows. Section presents related work. Section proposed RT-QoSFR scheme is introduced while Section explores simulation model, results, and discussions. Finally, Section provides the conclusions and identifies directions of future work.In OBS network, the exchange of data between the source and destination transfers through several sub-processes, which will either add some extra delay time or cause some data loss. These sub-processes can affect the QoS requirements and traffic contract in real time compressed traffic.Previously, several schemes in several OBS sub-process have been proposed to guarantee the QoS for the high priority traffic. Each scheme tried to achieve guarantee of QoS from different aspects. However, guaranteeing the QoS for real time compressed traffic over OBS has not been fully achieved.Tout) or when the number of bytes reaches the maximum value of Burst minimum (Bmin). Thus, this scheme is currently assumed to be the default burst assembly scheme. It combines the benefits of both the time-based burst assembly and the threshold-based schemes. However, the hybrid scheme is not considering the real time traffic delay requirements in the case of low network traffic load. Where, the real time traffic will be forced to wait until the timer reaches to its maximum value, then it is assembled and sent to the destination. This delay affects the real time traffic delay requirements.In the burst assembly sub-process, the hybrid time-and-threshold-based scheme  3] has  has 3] hOn the other hand, Learning-based Burst Assembly (LBA)  proposedIn the contention resolution sub-process, burst segmentation  is a conHowever, aggregating the burst from the high priority traffic only will increase the average of the loss of its packets. Therefore, a combination of several traffic types can be achieved in one burst. Researchers in  6] 7]  [7] 6] [  [7] 6] In the signalling sub-process, OBS network uses a one-way reservation mechanism to allocate the resources where the control packet proceeds the data burst in an amount of time called the ‘offset time’. The offset time is the amount of time required by the control packet to successfully allocate the resources. In case the offset time is not enough to allocate all the resources in the destination path, the data burst is dropped. The researcher in  proposedSong H, Brandt-Pearce M, Xie T, Wilson SG developed an innovative concatenation scheme that works in two TIER; (1) inner code, and (2) outer code. Inner code is being constrained code based on Total Impairment Extent Rank (TIER) while outer code being a low-density parity-check (LDPC) code. In order to avoid the affect of Amplified Spontaneous Emission (ASE) noise on system performance for long-haul fiber-optic communication systems, a novel TIER is developed to prevent deterministic physical impairments and the ASE noise. This TIER constrain code from coding scheme and restrain physical impairments, including linear effects and nonlinear effects .While, Song and Pearce  introducSame authors also worked on another model that works on several channel effects, fiber loss, frequency chirp, optical filtering, and photo detection which are omitted in the literature. The model offers an agreement with obtained results by split-step fourier simulation. Moreover, this model covers several characteristics such as ISI, inter channel interference, self-phase modulation, intra channel cross-phase modulation (XPM), intra channel four-wave mixing (FWM), XPM, and FWM to improve the system performance .The authors in  presenteOBS is considered as an optical network technique that allows wavelength-division multiplexing (DWDM) and in this regard, the method of Volterra series transfer function (VSTF) is presented that state characteristic coefficients to record intersymbol interference (ISI), self phase modulation (SPM), intrachannel cross phase modulation (IXPM), intrachannel four wave mixing (IFWM), cross phase modulation (XPM) and four wave mixing (FWM), to classify the influence of these components on the system output . FurtherThere is huge delay issues with real-time cloud services and in order to address these issues, Shojafar M, Cordeschi N, Baccarelli E  proposedAlternatively, an adaptive offset time scheme in OBS network  proposedIt is clear from the open literature that the preceding mechanisms are able only to guarantee either the delay requirements or data loss requirements. Furthermore, the fairness among the traffic type is absent in these schemes. Thus, a new scheme that guarantees both requirements with ensuring the fairness among the traffic types is required. This paper proposes a new scheme called Real Time Quality of Service with Fairness Ratio (RT-QoSFR).Tout) according to the traffic requirements and load so as to reduce the real time traffic packets loss, and at the same time, it guarantee the fairness for non real time traffic packets. The traffic ratio inside the burst is adapted based on a statistical study that has been carried out to find the best ratio (fairness ratio) for the real time traffic packets inside the data burst in various network traffic loads. Furthermore, Tout is adapted based on the most appropriate value to meet the end to end delay requirements of real time traffic. As a result, RT-QoSFR promises the required QoS that creates stability in the performance of the network, ensures the fairness between real time traffic, and non real time traffic, which all collaborate to provide a better QoS.In this section, a novel Real Time traffic—QoS with Fairness Ratio (RT-QoSFR) scheme based on a new burst assembly algorithm is introduced. RT-QoSFR adapts the burst assembly parameters  and the Real time traffic loss (Rloss).This statistical study has been carried out to find the best ratio (fairness ratio) for the real time traffic packets against the non real time traffic packets in various network traffic loads. The statistical study is based on the Significant difference (Sd) factors between the network’s Over all loss , two values of burst size, two different topologies , incremental load traffic rate, and ten ratio values for real time traffic (10-100%). As a result, ten Sd values have been produced for each case in the study as follows:x determines the real time traffic ratio in the data burst. In this equation, the value of x will be replaced, ten times, with the real time ratios in the data burst and stored in the Sd group. Consequently, based on these results, mathematical equations to find the range of the best ratio for real time traffic in the burst that can guarantee the fairness between real time traffic and non real time traffic have been found. The mathematical equations aim to find the lowest two minimum ratio in Sd; the Lowest ratio (Lowratio), and Second Lowest ratio (SLowratio). Firstly, the equation that aims to find the minimum Sd value  is as follows:Where Lowratio by searching in all the Sd group ratio and assign it to Lowratio as follows:Subsequently, it aims to find out the ratio of this value SLowratio) can be found by using a temporary group of elements Sdtemp that contains all Sd except Lowratio:The second minimum Sd  the OBS network configuration and (2) the real time traffic configuration. In the real time traffic configuration, CBR traffic and VBR traffic trace files have been created with incremental load. The results show that the aggregation process can be categorized based on the traffic load into three categories, which are the low, normal, and high loads. In the high load, the results show that the best ratio for real time traffic in the burst that can guarantee the fairness between real time traffic and non real time traffic is between 50% and 60%.Figs It is noted that the value of Sd gradually decreases from the ratio of 10% to 50%. This is because of the numerous number of data burst that is created and sent to the core network due to the small ratio of real time traffic in each burst. It leads to an increase the number of data burst needed to send all the real time traffic available in a certain period. This large number of burst increases the rate of burst loss in the core node which creates a discrepancy between the value of the network overall loss and the real time traffic packets loss. Thus, it can be noted that the Sd value goes high in the case of ratio of 10% and reduces with the increase of the ratio of real time traffic in the burst which reduces the rate of burst loss until it reaches to the ratio 50%.In contrast, it is noted that the value of Sd gradually increases from 60% up to 100%, due to the rise of the rate of real time traffic packets loss caused by its high ratio in the burst and the low rate of overall loss which is created by the low number of bursts. Therefore, the Sd value is high at these ratios and getting a raise with the increase of the ratio of real time traffic in the burst. It causes an increasing of the rate of real time traffic packets loss until it reaches to the ratio 100%. In the normal load, the results show that the best ratio for real time traffic in the burst that can guarantee the fairness between real time traffic and non real time traffic is between 30% and 40%. Figs In normal load case the value of Sd is low for the low ratios (10–50%) and high for the high ratios compared with the high load traffic case. This variation is due to the traffic load which increases the rate of traffic load in the high load case; while the rate of real time traffic packets loss is similar to both cases which make the Sd value goes high in the high traffic load case. In contrast, for the high ratios from 50% up to 100%, it is noted that the value of Sd is higher than the values of the low ratios due to the decrease of overall rate loss. In this case, the rate of real time traffic packets loss is similar to both normal and high traffic load case, which makes the Sd value higher in the normal traffic load case.In the low traffic load, the findings show that the best ratio for real time traffic in the burst that can guarantee the fairness between real time and non real time traffic ranges between 10% and 20%. Figs On the contrary, it is noted from In the following section, the design of RT-QoSFR scheme is discussed, therefore, Tout based on the maximum packet transfer delay value. The timers are separated due to the needs for a differentiated service, as well as to make the default internal timer works for all traffic types in case of no real time traffic. RT-QoS scheme starts with the first phase to identify the network traffic type and traffic load. RT-QoS scheme studies and analyzes the network load in each second through the following steps: it gets the summation of the Aggregation time for each burst (Atime) every one second and assigns it as a Total time value (Ttime) according to RT-QoSFR scheme works inside the optical network edge node to classify both traffic load and traffic type. RT-QoSFR scheme differentiates the data traffic based on the destination into different queues. In each destination queue, RT-QoSFR also differentiates the data traffic based on the traffic type into real time traffic or non-real time traffic queue. RT-QoS scheme employs two timers to aggregate the data; a default internal timer, and a separate timer on the real time traffic queue. The real time traffic timer is used to calculate the appropriate burst assembly time values Bsize) is converted into bits and assigned to the Temporary size Tempsize variable. Next, RT-QoS checks the Ttime value, if it becomes 1 second, it will assign the Tempsize to the total burst size Tsize, otherwise, it continues the counting. Then, the summation of the Burst size (Lavg:After getting the number of bits in one second by using RTTtimer). The delay requirement for real time traffic is provided with the packets QoS parameters. The packet maximum acceptable transfer time over the network can be determined by MaxDelay. In CBR and VBR traffic, the packet maximum transfer time over the network is clearly stated in a parameter called Maximum Cell Transfer Delay (MaxCTD). Thus, to fulfil the real time traffic QoS delay requirements over OBS, the MaxDelay parameter must not be less than the OBS entire delay, i.e.,If the traffic type is real time traffic, the system will use Real Time Traffic timer , and the Propagation delay (Pdelay).In the OBS network, the entire delay is the collection of the Burst Assembly time ODelay value is variable such that it can be used for any delay in other domains than OBS, and p is any potential value. As a result, the MaxDelay must be greater than or equal to these parameters, so,However, to guarantee the real time traffic end to end delay required, the delay in other domains must be considered. Thus it is assumed that, Other domains Delay Pdelay is calculated based on the distance between the source and the destination (d) and the wavelength propagation speed (s) as follows:In the OBS networks, h is the number of nodes between the source and the destination, PT is the processing time and ST is the switching time of the core node. By compensating Eqs , then the system will change the Maxdelay value to the new required one, which is the new MaxCTD value. Besides, RT-QoSFR will aggregate and send the current burst which was configured based on the previous delay requirements to ensure that each data packet has been met with its delay requirements. Then, the value of the Real Time Traffic timer (RTTtimer) is calculated based on the new value as follows:As can be seen from Tout,The real time traffic timer value after that is assigned to the MaxCTD is not less than Maxdelay, then the system continues with the previous delay requirements. Afterwards, all packets that arrived during this period will be queued at the destination queue. RT-QoSFR checks whether the Tout or Bmax reaches its maximum value to send the burst, it will determine the network traffic load to adapt the data burst aggregation ratio according to the network traffic load and send the data burst. After getting the traffic type, RT-QoS distinguishes the network traffic load into three categories according to the network load which are the high load, normal load, and low load. Based on the network load, RT-QoSFR scheme will adapt the ratio of the real time traffic inside the burst. The ratio of the real time traffic packets will be set to an initial ratio plus membership value for every category. After obtaining the network traffic load rates, RT-QoSFR will determine the traffic load category to be high, normal, or low traffic load. Then, RT-QoSFR will find the Membership value  for each traffic load category as follows:Lavg determines the network traffic load rate per 1 second, Hload represents the value of the beginning  of the high traffic load, Arang stands for the maximum value of the membership value, and Lload determines the highest value of the low traffic load parameters. After that, RT-QoSFR will find the Real Time Traffic average (RTTavg) in the burst by adding the MValue to the ratio of real time traffic inside the burst based on the network traffic load. So, the RTTavg will be calculated as,Havg, Lowavg and Navg determine the ratios of real time traffic inside the burst for high between (50—60%), low between (10—20%) and normal between (30—40%) traffic load parameters, Arange stands for the maximum value of the membership value. After getting the average of real time traffic in the burst, RT-QoSFR will find the total number of bytes for the real time traffic (RTTBytes) and the non real time traffic (NRTTBytes) in the burst as the following,Bsize is the summation of the Burst size. The next step, RT-QoSFR will aggregate the burst with RTTBytes from the real time traffic queue and NRTTBytes from the non-real time traffic queue. Finally, the system will check if there is a continuous flow of data. If this happens, the system will repeat the same procedures.On the other hand, if the new value of In this section, the simulation scenarios that have been used in this paper are introduced. The simulation has used NCTUns simulator to develop, evaluate the performance of proposed schemes and compare them with other schemes. The proposed schemes have been studied using the simulation model with two types of traffic , four values of MaxCTD, two value of burst size, and two different topologies , low load (512kb), and bursty load. BARDR scheme has been studied under all the possible traffic load rates as mentioned above to ensure the guarantee of real traffic delay requirements within any load. μs, 100 μs, 70 μs, and 125 μs. These values have been assumed in this range in order to get an excellent quality of data.For all traffic loads, three MaxCTD values have been assumed to be 92 The RT-QoSFR scheme has been evaluated using a customized simulation model with several traffic loads, two types of traffic (Constant Bit Rate (CBR), Variable Bitrate (VBR)), and NSFNET topology as shown in It can be noted from Observing In order to show the ability of RT-QoSFR to ensure the fairness for non-real time traffic, RT-QoSFR has been compared with Composite Burst Segmentation (CBS) and Non-CBS (NCBS) for several real time traffic ratios of 80, 50, and 20%. The burst segmentation scheme has been chosen in this comparison due to the results shows that it is the second lowest loss rate in the previous comparison. The results also show that RT-QoSFR scheme guarantees the fairness for non real time traffic packets.In the NCBS scheme, the results show that RT-QoSFR can guarantee the real time traffic packet loss, at the same time guarantee the fairness for non real time traffic packets, which lead to the network performance stability. Figs The packets loss rate comparison evaluates the overall packets loss of the network, and the real time traffic packets loss for RT-QoSFR and NCBS schemes. The results clarify that the NCBS scheme reduces the overall loss in general. However, it does not guarantee the packet loss for the real time traffic which is high.Figs On the other hand, the results in Figs In contrast, the results in RT-QoSFR series show that the rate of loss for real time traffic packets is higher than CBS in the case of 20%. However, the stability in the performance of the network and the fairness among real time traffic and non real time traffic leads to provide a better QoS to the real time traffic packets. Moreover, the overall packet loss rate in RT-QoSFR series is less than CBS in the case in 20%. This also reduces the total number of lost packets from the real time traffic.MaxCTD value unlike the other schemes, which exceed the MaxCTD value in the low load rate. Moreover, it shows that RT-QoSFR scheme guarantees the traffic delay requirements in all traffic load cases. Furthermore, it guarantees the delay requirements, even in the presence of a contention topology.For the real time traffic delay requirements, the results show that the entire delay using RT-QoSFR does not exceed the MaxCTD value, which equals to 100 μs. This value represents the maximum packet delay in this simulation model. On the other hand, the average packet delays for other schemes are exceeding the MaxCTD value for the other schemes. This is because the rate of low load traffic, normally, is not enough to create a burst within the interval time specified in the timer. Consequently, the data packets have to wait in order to aggregate enough data or until the Tout parameter reaches its maximum value, this leads not to fulfil the traffic delay requirements.In the low offered load, it is noticed that the RT-QoSFR scheme delay is less than the Tout parameter based on the value of MaxCTD. This process led to allow the data packets to be sent before its maximum delay time. Therefore, for RT-QoSFR scheme the average packet delay does not exceed the maximum packet delay level as shown in RT-QoSFR scheme, in case of low load traffic, can guarantee the packet delay requirements by setting the value of In contrast, in the high load traffic, the amount of data is enough to create a burst before the interval timer value reaches its maximum value. Thus, the burst will be sent in a time that is lesser than the maximum packet delay. Consequently, the RT-QoSFR scheme delay curve is convergent with the other schemes because the timers will not work in this case and the burst will be sent based on the amount of traffic, which is already equal. In the case of VBR traffic shown in In the case of CBR traffic, the convergent with the curves of RT-QoSFR and other schemes occurs after the load of 60%. While, in VBR case the convergent starts at the load of 80% due to the different sizes of packets in VBR traffic. Generally, The results show that RT-QoSFR scheme can reduce the real time traffic packets loss, at the same time guarantee the fairness for non real time traffic packets, and guarantee the delay requirements for the real time traffic.In this section, a simulation model has been developed to study RT-QoSFR scheme with Simple four-node OBS (SOBS) topology to prove the ability of RT-QoSFR scheme to work with various network topologies and nodes.μs from the load of 10% to the load 50%. In the high traffic load, the average packet delay is lesser than the required maximum delay. Figs The results show that RT-QoSFR scheme is able to guarantee the delay requirements with various network topologies and nodes. It is noted that RT-QoSFR scheme was able to guarantee the average packet delay with the required MaxCTD. The average packet delay does not exceed the value of 100This paper has proposed a novel RT-QoSFR scheme that can adapt the burst assembly parameters according to the traffic needs to guarantee the real time traffic requirements and ensure the fairness for the other network traffics.RT-QoSFR has classified the network traffic load into three categories which is utilized to adapt the ratio of the real time traffic inside the burst for reducing the real time traffic packets loss and guaranteeing the fairness for non real time traffic packets. The fairness ratio for the real time traffic packets in the burst have been found to be 50—60%, 30—40%, and 10—20% for high, normal, and low traffic loads, respectively.MaxCTD parameter value in the real time traffic. Furthermore, it can reduce the real time traffic packets loss and guarantee the fairness for non real time traffic packets. Moreover, RT-QoSFR guarantees the stability in the performance of the network, the delay requirements, and ensures the fairness between real time traffic and non real time traffic, which provides a better QoS.The results show that RT-QoSFR can guarantee the entire delay of OBS network such that it does not exceed the


This article describes the progress in the development of the atomic force microscope as an imaging tool and a force transducer, with particular reference to applications in food science. Use as an imaging tool has matured and emphasis is placed on the novel insights gained from the use of the technique to study food macromolecules and food colloids, and the subsequent applications of this new knowledge in food science. Use as a force transducer is still emerging and greater emphasis is given on the methodology and analysis. Where available, applications of force measurements between molecules or between larger colloidal particles are discussed, where they have led to new insights or solved problems related to food science. The future prospects of the technique in imaging or through force measurements are discussed.  •Ultrastructure of polysaccharides.•Structure-function relations of food molecules.•Physical effects of interfacial stabilisers and deformable colloids.•Bio and physical activity of pectin components.•Characterisation of glycan distribution on mucin. During his extensive research career there has been substantial progress in the understanding of food structure, partly due to the development and application of new experimental methods. Prof Phillips has contributed greatly to the dissemination, development, and application of this knowledge through his own research group, the origination and editorship of the Food Hydrocolloids journal, and the initiation and running of the highly successful Gums and Stabilisers & International Food Colloids meetings. In celebration of this contribution this article reviews developments in the use of one of these new techniques, namely atomic force microscopy (AFM), both as a microscope and as a force transducer. This is a technique we have been intimately involved with over this period. Since its inception in the 1980s AFM has led to considerable insights into a range of food structures. The review will focus on uses in food science and on new developments and, in particular, highlight the increasing use of AFM as a force transducer, which is likely to open up new areas of understanding in the near future.This review is dedicated to Prof. G. O Phillips on the occasion of his 902An atomic force microscope (AFM) scans a tiny and extremely sharp tip that is mounted on the end of a flexible cantilever over the surface of samples – it is similar to the action of a stylus on a record player, but in terms of microscopy effectively a nano-profilometer. Unlike all other forms of microscopy it has no lenses and does not image the sample by ‘viewing’, rather it does so by ‘feeling’ the surface of the sample .2.1The development of atomic force microscopy (AFM) as a tool for imaging biological systems offered considerable promise for imaging at molecular, or sub-molecular level, in a liquid environment . Realisa2.1.1AFM has provided new information on molecular size and shape under more natural conditions. For helix-forming polysaccharides, which function as gelling and thickening agents, it has not only been possible to characterise the molecules but to investigate their functional role. Gellan gum is a model system for investigating polysaccharide gelation, and through the imaging of the molecules, gel pre-cursers and the surface of intact hydrated gels it has been possible to visualise the molecular networks formed in the gels and infer the mode of association . MicrogeMicroscopic techniques such as AFM allow the characterisation of heterogeneity at the molecular level. An example is the observation of branching in semi-flexible molecules such as arabinogalactans . An analAFM images revealed for the first time irregular low level branching 2.1.2The internal structure of more complex biological structures can be investigated by imaging flat cut surfaces or sections of the samples: examples include the study of plant cell walls or starch granule structure .In order to investigate the internal structure of starch it was found necessary to embed isolated starch granules in non-penetrating resins, in order to avoid artifacts induced by penetrating resins , and to The structure and changes in structure of air-water or oil-water interfaces is of crucial importance for understanding the texture, stability and digestion of food foams or emulsions. The use of AFM has led to important new insights into the behaviour of interfaces .AFM proved particularly useful for studying protein networks at interfaces, which were shown to form elastic interfacial networks . The intBecause the proteins are linked into a network it is not possible for the surfactant to displace individual proteins. Rather the surfactant targets the Achilles’ heel of the network – the holes or defects. Increasing surfactant adsorption at the interface leads to growth of surfactant domains  confirmed that the larger surfactant domains observed by AFM, prior to collapse of the protein network, were seen directly at the liquid interface by BAM . StudiesAn understanding of protein-surfactant interactions at interfaces provides a basis for improving, rationally the stability of the interface, and hence the lifetime of the product. However, the methodology can be extended to investigate the changes in interfacial structure during digestion. Experimental studies on the effect of digestion conditions on interfacial structures were initially made on model air-water systems, and then extended to oil-water interfaces as more realistic models of food emulsions.in vitro duodenum conditions show that displacement occurs via an orogenic mechanism  generated small changes in structure but, unexpectedly combined effects of pH and temperature weakened the networks which, never-the-less, still remained intact. For oil-water interfaces the effects of gastric media on pre-formed β-lactoglobulin layers adsorbed at two different oils (tetradecane and olive oil)-water interfaces showed the importance of the nature of the oil phase on network stability and that protein unfolding, induced by the oil phase, may offset certain aspects of the weakening of the networks induced under gastric conditions .Pepsin was used to investigate exposure of interfaces to proteolysis in the stomach. Understanding protein unfolding on adsorption and under gastric conditions proved important. At air-water interfaces the action of pepsin under gastric conditions on pre-formed β-lactoglobulin networks led to partial hydrolysis of all surface-adsorbed proteins: however, proteolysis did not disrupt the protein network which remained intact and capable of resisting surfactant displacement. Surfactants may be present under gastric conditions and, since orogenic displacement causes enhanced exposure of protein structure to the aqueous media, their effects on proteolysis were studied. Modification of the surface conformation of the proteins during ‘orogenic’ displacement led to an unexpected synergism, which enhanced proteolysis. However, importantly, strengthening of protein networks to inhibit surfactant domain growth should thus not only restrict bile salt adsorption, but also inhibit proteolysis in the stomach. Such observations are important new generic features of digestion, which can be manipulated for rational design of food structures to promote health.Studies at air-water interfaces were extended to oil-water interfaces and emulsions. The generic features were retained at the oil-water interfaces and the nature of the oil phase is of importance. Studies at interfaces were extended to study effects of gastric digestion on protein-stabilised oil-water emulsions: linking basic studies to realistic models. Interestingly, digestibility profiles of interfacial proteins depended on the nature of the oil phase. The type of oil affects the surface conformation of the protein affecting proteolysis in the stomach. Proteomic analysis of the peptides generated during the digestion process provide new information on proteolysis by pepsin under gastric conditions, as well as novel information about the interfacial properties and conformation of β-lactoglobulin adsorbed at different oil-water interfaces .In general polysaccharides are poor emulsifiers. However, certain protein-polysaccharide complexes, such as gum Arabic and sugar beet pectin, show emulsifying properties, attributable to the protein component. In the case of sugar beet pectin AFM studies provided insights into the role of the protein. AFM images b of sugaIn order to test this model for emulsification AFM was used to image the structures formed at interfaces , and the33.1P. The most commonly applied formula describes the extension, z, of a worm-like chain with contour length Lc and persistence length, lP, in response to a stretching force, F, as;B, is Boltzmann’s constant and T, absolute temperature =LC[Another aspect of retraction force measurement is exploration of ligand-receptor interactions by functionalising the AFM tips . Tip functionalisation protocol includes use of a heterobifunctional spacer molecule, esterified polyethylene glycol (PEG) . It has Force–distance curves have been successfully modelled in a recent ground-breaking study that demonstrated that deformable colloids such as oil droplets and air bubbles are particularly susceptible to studies by AFM because they adapt their shapes in response to the changing forces caused by the non-adsorbed species  in the closing gap . This op3.1.1The potential bioactivity of polysaccharides has been recognized for many years, particularly in the area of dietary fibre and the beneficial effects this material has on gut health . Recent One of the main molecules which protects the gastrointestinal tract is mucin; a glycoprotein composed of a polypeptide backbone with a very large number of glycan sidechains. The mucin molecular mass is generally composed of >50% and often 70–80% carbohydrate. Mucin is a highly heterogeneous polymer in terms of its glycan sidechains . The sidDespite the inherent heterogeneity of the glycan substitution on mucins, the fact that this technique was able to discriminate mucins from different regions of the gut and track changes induced by enzymatic attack holds much promise. AFM can, not only begin to read the glycocode on mucin, but also investigate which external factors (dietary components such as pre-biotics and pro-biotics) may re-write it.3.1.2A new technique for direct examination of colloidal interactions was provided by AFM . This foγ is the surface tension and r the droplet radius. Thus adsorption of protein should lower the surface tension, lowering the internal pressure making the droplets more deformable.Oil droplets can be emulsified by two different compounds; surfactants and proteins. As illustrated in The measurement of deformability of soft materials by AFM is quantified by the variation of the deflection of the cantilever as the sample is pressed against a rigid surface. The deformation of a soft sample will reduce the deflection of the cantilever compared to that seen for the cantilever alone. As droplets collide the nature of the interface will determine the flow of liquid from between the converging droplets. Force spectroscopy can be used to monitor differences in the hydrodynamic flow of the thin liquid film (TLF) that becomes forced out as the droplets are pushed together as illustrated in Force spectroscopy of droplet-droplet interactions were obtained by positioning the cantilever-droplet assembly directly over another droplet of approximately equal size, which was bound to the glass slide. Each measurement consisted of a force versus distance cycle in which the droplets  were first pushed together, and then pulled apart. Relaxation processes were captured by imposing a fixed dwell time following the approach part of the force versus distance cycle, once the cantilever deflection had reached a predetermined value of loading force, termed ‘trigger force’. During the dwell period the feedback loop of the AFM holds the z stage of the piezoelectric scanner in position (using a second feedback loop to eliminate any piezoelectric creep) and deflection of the cantilever is monitored with time.The experiments provided direct measurement of the drainage of the thin liquid film. The data in The hydrophilic component of the protein in the aqueous phase can slow the flow of the water molecules as the droplets are pushed together but more data in the study revealed that the flow rate slows further as they become closer . Two facThe force spectroscopy measurements can be used to extend studies of droplet-droplet interaction for systems where the interfacial structure is well defined. An example is the case of droplets stabilised with sugar beet pectin (SBP) extracts . Sugar bHigher concentrations of each of the polymers in the bulk phase (0.5%) caused hysteresis between the approach and retract phases . This reFurther confirmation that it was due to the structural correlation of the liquid film separating the droplets was provided by varying the polymer concentrations and also the velocity of the droplets being pushed together and pulled apart . The depAs the PSS concentration was increased further the effects became more pronounced b and theThe longer the time duration following jump-in, the larger is the subsequent pull-off required to separate the droplets. The interpretation of these effects is that they result from the formation of a region devoid of polymers in the thin liquid film between the drops . Neutron scattering studies have provided experimental evidence that such ‘black films’ can also occur between adhesive emulsion droplets . Fig. 19The final examination of the effect of SBP emulsified droplets was the potential for partial coalescence as the pectin chains can link together in certain conditions forming coagulated strings of the droplets which can have a significant effect on the physical properties of such emulsions. Normally the extension of the pectin chains into the bulk aqueous phase will lead to a steric repulsion between droplets as proposed by . ManipulFinally, another significant set of AFM colloidal studies have been carried out at the University of Melbourne. In addition to the AFM force measurements they have created mathematical models of most of the aspects; deformability effects , velocit4The use AFM as a microscopic tool is now used widely to tackle problems in food and biological science. A key advantage is the ability to observe structure and structural heterogeneity at the molecular level, under ‘near natural’ conditions. Although widely used, the method is still not routine, and methodologies need to be modified or developed to suit particular biological samples. Use has led to new understanding of complex food systems with resultant applications. The use of the AFM as a force transducer is emerging as a new tool in food science, which can assist and expand the role as a microscope. For example whereas interactions between larger molecules such as proteins and polysaccharides can be visualised, force measurements are needed to identify and characterise binding of smaller molecules such as in oligosaccharide-protein interactions, which could not be seen directly by AFM. Similarly the ability to probe interactions between large objects, such as air bubbles or oil droplets offers the possibility to extend the imaging and understanding of interfaces into investigating their effect on the pair-pair interactions responsible for foam or emulsion stability. In addition, given an understanding of the changes in interfacial structure during complex processes such as digestion it ought to be possible to determine their effect on the stability and breakdown of food structures on consumption. Thus the combined use of the techniques offers routes to the design of novel food structures to tackle problems in health, dietary choice and nutrition.


The behaviour of a granular material is mainly dependent on its frictional properties, angle of internal friction, and cohesion, which, together with material density, are the key factors to be considered during the scaling procedure of analogue models. The frictional properties of a granular material are usually investigated by means of technical instruments such as a Hubbert-type apparatus and ring shear testers, which allow for investigating the response of the tested material to a wide range of applied stresses. Here we explore the possibility to determine material properties by means of different empirical methods applied to mixtures of quartz and K-feldspar sand. Empirical methods exhibit the great advantage of measuring the properties of a certain analogue material under the experimental conditions, which are strongly sensitive to the handling techniques. Finally, the results obtained from the empirical methods have been compared with ring shear tests carried out on the same materials, which show a satisfactory agreement with those determined empirically. Analogue modelling represents one of the most useful tools to test hypotheses on the evolution and controlling factors of geological processes at different length and time scales. One of the fundamental steps is to identify the proper analogue material to replicate natural rocks and model the natural process under investigation.Although the behaviour of natural rock is complex  is thus frictional shear failure, governed by the Mohr-Coulomb criterion of failure (Equation (1)) describing the linear increase in the strength of rocks with depth:fficient ,4, and λtic load . On the tic load  proposedFollowing the above concepts, analogue modellers commonly use different dry granular materials  to simulate upper crustal deformation. Their behaviour is generally assumed to follow the Coulomb failure criterion, with constant frictional properties and is strain-rate independent , therebC) and (2) the bulk density.The main material parameters that are necessary to be known for characterizing a granular material for analogue modelling purposes are: (1) the frictional parameters  of the straight line obtained by a linear regression of shear test results. However, this method may give values that are not fully realistic since it contradicts the common observation that dry granular materials are virtually cohesionless )Cohesive granular materials are generally silica powder ,40, fineTo obtain new analogue granular materials that would combine the advantages of quartz sand and cohesive materials, we mixed quartz sand and K-feldspar powder in different proportions. Our aim is to obtain a material characterized by a proper internal friction angle able to simulate crustal rocks, with moderate cohesion, as well as a suitable grain size allowing the development of numerous and detailed structures for a complete fracture/fault  analysis.We tested quartz sand  and K-feldspar fine sand  and different mixtures between these two materials . The useFontainebleau Q-sand  is a welThe grains have mostly an angular shape, and are mainly composed of potassium feldspar (83%), with subordinate sodium feldspar (6%), quartz (10%), and a smaller part of kaolinite (1%).As mentioned earlier, granular material properties are strongly sensitive to the handling techniques ,9,19: poσ1 and the new fault plane is given as θ = 45° − Φ/2. The angle of internal friction was determined by the means of two different empirical methods widely used in the literature: (1) by measuring the angle of repose ,43,44,45The experiments were built with a simple set-up : a 6 cm-thick layer of sand mixture with a thin colored layer (made of the same sand mixture) every 1 cm of thickness to help recognizing the faults. The sand is colored with Indian ink and naturally dried, over a period of days. Coloring sand is a very common practice in analogue modeling, and it is recognized to not significantly influence the mechanical properties of sand. The sand pack was 20 cm long and rested on the base of a deforming pure/simple shear apparatus  and model deformation invariably consisted of a couple of conjugate faults developed above the velocity discontinuity. Extension was applied until the first extensional faults were clearly observable on the mode surface. The deformation distribution at grain size level is sensitive to the material properties, implying that the amount of extension needed for the first faults to develop may slightly vary depending on the model. In any case, the bulk extension was of the order of a few millimeters (2–4 mm). This however does not influence the model results. Once the experiment was stopped, the model was wetted to allow for cutting model cross sections in the model central part. The fault dip was measured directly above the velocity discontinuity, at the model base, where the conjugate faults initiate and have a rectilinear attitude. In this location, the vertical normal stress corresponds to the maximum principal stress 1 .α . The value of the internal friction coefficient calculated with the above-described method is sensitive to the handling technique; particularly the more densely packed sifted sand exhibits a higher friction coefficient than poured sand 19]).As mentioned above, the angle of internal friction when measured with the fault dip methods represents the peak value for the experimental conditions; this explains why this value is always higher with respect to the angle of internal friction calculated considering the critical angle of repose . On the The obtained values for pure Q-sand are well in the range of previous measurements . A progressive increase in the content of the less dense K-feldspar in the mixture should result in a progressive decrease of the mixture bulk density itself. However, we should consider the different grain size of the two pure terms: part of the finest grains of the K-feldspar fill the intergranular voids in between the larger grains of the Q-sand, with no increase in volume. This implies that adding a small quantity of K-feldspar sand to Q-sand results in an increase in the bulk density of the mixture, compared to the pure Q-sand (3). The mixture bulk density tends to slightly decrease with increasing the K-feldspar fraction (>10% in weight); for K-feldspar content higher than 30% in weight, the decrease is more pronounced and tends to become linear.The obtained values of density are reported in e Q-sand . As a ma0 and V500. The Hausner ratio (and the volume variation percentage) linearly increases with the K-feldspar content, in a manner comparable with the decrease in density observed above. However, the small increase in density that we observe with small K-feldspar content is not reflected in a small decrease in the Hausner ratio value with an equal mixture composition.The Hausner ratio values are shown in Pure Q-sand, with large, rounded, and poorly-graded grains, is thus characterized by a low Hausner ratio . Moreover, the use of a vibrating device to compact a mixture of different sands could lead to separating the different components, and its use should thus be carefully evaluated when working with sand mixtures.The use of higher K-feldspar mixtures may yield the disadvantage of having dishomogeneities internal to the analogue model, even though high-cohesion materials offer the advantage of suitably reproducing specific geological settings and processes, including those with relatively high length scaling factors , and the cohesion was accordingly calculated using the two distinct angles of internal friction values (see Equations (3) and (4)) . ObserviThe obtained values are very low, and the cohesion is mostly negligible in the examined granular materials. As for the angle of internal friction, we can observe a clear increase in cohesion with increasing the K-feldspar content in the mixture.The lower value is found for the pure Q-sand, for which the cohesion is completely neglectable; the observed vertical wall is probably the result of a few grains stacked one on the other and there are no inter-particle forces playing a detectable role. The cohesion obtained from peak friction is maximum for the quartz 60-K-feldspar 40 mixture and then slightly decreases for higher K-feldspar content. The observed increase with K-feldspar is probably related to the higher fine fraction in the mixture, in that the cohesion is the expression of the inter-particle forces and is consequently sensitive to the grain shape and grain size distribution e.g., ,44). Sim. Sim44])Our results indicate that the cohesion of a granular material is also a function of the density and the degree of compaction . In cconτ) and the rotation into displacement and plot those as shear curves. Samples were placed into the shear cell in the most similar way that we used for the empirical methods and analogue models, and measurements were carried out for different normal loads. It should be said, however, that given the conformation of the apparatus, it is not possible to exactly reproduce the same handling techniques and experimental conditions. To test the reproducibility, each measurement was repeated three times per normal load. Ring shear tests were performed at a constant velocity of 0.5 mm/min with a normal stress ranging from 1.8 to 4.4 kPa (shear stress range from 1.3 to 5.6 kPa). For comparison, the results obtained for the pure Q- and K-feldspar sands are reported in To test the empirical approach, the mechanical properties of different granular mixtures of Q-sand and K-Feld sand were also determined using a ring-shear tester at the Engineering Geology Laboratory of the University of Florence. The apparatus consists of an annular shear cell containing the sample and a lid that is pressed onto the sample at a preset normal load which is kept constant throughout the experiments . Each exTo characterise the properties of granular materials used in analogue modelling, modellers commonly use classical technical instruments such as Hubbert-type apparatuses and ring shear testers, which allow investigating the response of the tested material to a wide range of applied stresses. However, analogue modelling laboratories do not always have direct and easy availability of all the equipment needed to carry out these experiments, and above all, these devices often do not allow the characterization of the materials at the experimental conditions, which are affected by particular handling techniques and procedures. In this study, we have explored the possibility to determine the material properties by means of different empirical methods applied to mixtures of quartz- and K-feldspar sand prepared at the experimental conditions. We tested their reliability and we suggested that the adopted empirical methods could represent an easy, cheap, and fast tool for determining the properties of analogue brittle materials, even if, regarding the cohesion evaluation, it is possible that there is an underestimation in comparison to the instrumental measurement techniques that commonly report higher values. Nevertheless, we should consider that by using the most common instrumental techniques  and by assuming a linear relationship between the shear stress and normal stress, the cohesion is inferred as the intercept with the shear axis, in a shear stress-normal stress plot. Cohesion values obtained with this method are higher (up to some hundreds of Pa; e.g., ,19,29,39


Getting a good feature representation of data is paramount for Human Activity Recognition (HAR) using wearable sensors. An increasing number of feature learning approaches—in particular deep-learning based—have been proposed to extract an effective feature representation by analyzing large amounts of data. However, getting an objective interpretation of their performances faces two problems: the lack of a baseline evaluation setup, which makes a strict comparison between them impossible, and the insufficiency of implementation details, which can hinder their use. In this paper, we attempt to address both issues: we firstly propose an evaluation framework allowing a rigorous comparison of features extracted by different methods, and use it to carry out extensive experiments with state-of-the-art feature learning approaches. We then provide all the codes and implementation details to make both the reproduction of the results reported in this paper and the re-use of our framework easier for other researchers. Our studies carried out on the OPPORTUNITY and UniMiB-SHAR datasets highlight the effectiveness of hybrid deep-learning architectures involving convolutional and Long-Short-Term-Memory (LSTM) to obtain features characterising both short- and long-term time dependencies in the data. Human Activity Recognition (HAR) is a research topic which has attracted an increasing amount of attention from the research community, in the wake of the development and spread of increasingly powerful and affordable mobile devices or wearable sensors. The main goal of HAR is automatic detection and recognition of activities from the analysis of data acquired by sensors. HAR finds potential applications in numerous areas, ranging from surveillance tasks for security, to assistive living and improvement of the quality of life or gaming ,2.Activity Recognition Chain (ARC) [Data acquisition: Choice and setup of wearable sensors. Constraints on the choice of sensors due to the nature of activities to be recognized, or potential implementation-related issues  have toData pre-processing: Operations to make the data suitable for further analysis. They comprise various techniques including sensor calibration, unit conversion, normalization or cleaning of corrupted data which could be caused by possible hardware failures or problems on data transfer. In cases involving the use of several sensor modalities, the synchronization of different sensor channels also has to be considered.Segmentation: Identification of segments which contain information about the activities to be recognized. This step serves two main purposes. It filters the data by excluding parts irrelevant to the recognition problem. Additionally, it reduces the quantity of data to be processed by the subsequent stages of the ARC by extracting segments of limited size. This last point especially matters in practice, as hardware-related constraints can limit the amount of data which are possible to process at each time step.Feature extraction: Computation of values to abstract each segment of data into a representation relevant to the activity associated with this segment. Finding a high-level representation is important to ensure the generalization capacity of the HAR system on large-scale data.Classification: Construction of a classifier using the extracted features. The classifier provides separations as clear as possible between different activities in the feature space, i.e., to establish rules to distinguish between the features computed on segments associated with one class and the other segments.The development of a HAR system typically follows a standardized sequence of actions involving the use of sensor engineering, data analysis and machine learning techniques. in (ARC) —which coExperience has shown the final activity recognition performances to be very dependent on every step of the ARC. While many extensive theoretical studies have also been carried out in the past decades to establish powerful—and nowadays standardized—classification models such as Support Vector Machine (SVM)  or softmfeature learning techniques allowing models to learn automatically features from data with minimal or even no manual input. Solutions using Artificial Neural Networks (ANNs) have especially exploded in popularity in the past few years. ANNs are composed of connected non-linear computational units called artificial neurons, organized in a layer-wise structure, with each layer being composed of an ensemble of neurons. Each neuron of a layer encodes a specific feature computed on the outputs of the previous layer; a behavior particularly highlighted for image classification [hidden (intermediate) layers) can craft highly abstract features by stacking hidden layers, which enables the computation of features on the low-level descriptors encoded by the first layers of the network. The ways to re-use learned features are also increased by allowing a higher number of possible connections between input and output neurons [Notable progress has been done recently to find fication . Deep AN neurons . The eff neurons . Consequ neurons ,12,13, aHowever, despite the promising results obtained, it remains actually difficult to assess rigorously the efficiency of feature learning methods, especially deep-learning related ones. The lack of details about the pre-processing of data —sometimes even about the implementation of the feature learning process itself —can make the reproduction of the reported performances complicated. It is also difficult to compare the different approaches because of a lack of standardization regarding the ARC steps: differences in the benchmark HAR dataset(s), data pre-processing operations, segmentation techniques and parameters, or classification models are all obstacles hindering this goal.The main contribution of this paper is to tackle both of the aforementioned issues regarding the evaluation of feature learning models for HAR. To address the first problem concerning the lack of standardized framework, we define our own evaluation framework where all stages of the ARC— except the feature extraction step—are fixed (as shown in http://www.pr.informatik.uni-siegen.de/en/human-activity-recognition). In addition, a generic evaluation script is made available. It can be re-used by other researchers to rigorously evaluate the performances of their own feature learning approaches on the datasets we used, in such a way that allows direct comparisons with our own results.To address the second issue regarding the lack of implementation details, we provide all the datasets, codes and details on how to use them needed to make the reproduction of our results as easy as possible  comprising simple statistical metrics computed on data: this approach constitutes a baseline in our comparative study as the only manual feature crafting method.Multi-Layer-Perceptron (MLP): the most basic type of ANN featuring fully-connected layers. The features learned by this model are obtained in a supervised way. The MLP results are used as a baseline for automatic supervised feature crafting.Convolutional Neural Network (CNN): a class of ANN featuring convolutional layers which contain neurons performing convolution products on small patches of the input map of the layer, thus extracting features carrying information about local patterns. Apart from image processing , audio rLong Short-Term Memory network (LSTM): one of the most successful and widespread variant of Recurrent Neural Networks which feature layers containing LSTM cells, able to store information over time in an internal memory. LSTM networks are used to capture temporal dependencies in diversified application fields such as automatic translation , image cHybrid model featuring CNN and LSTM layers: taking advantage of the high modularity of ANN-based architectures, previous studies in sensor-based HAR reported that hybrid architectures can extract features carrying information about short and long-term time dependencies, and yield better performances than pure CNNs or LSTM networks .Autoencoder (AE): a class of ANNs trained in a fully unsupervised way to learn a condensed representation which leads to the most accurate reproduction of its input data on its output layer. The results obtained by this approach are used as a baseline for unsupervised feature learning.codewords—of the signals used for the learning. The ensemble of codewords (codebook) is then used to extract histogram-based features based on similarities between subsequences of the processed data and codewords. Codebook-based methods can be regarded as one-layer CNN, as codewords are learned in a similar unsupervised way to convolutional kernels. They were used in previous works for time series classification [Codebook approach (CB): an unsupervised feature learning method based on the determination of “representative” subsequences— referred to as fication  or HAR [fication .This section presents succinctly the feature learning approaches selected in our comparative study. We test seven different feature crafting approaches covering most of the recent state-of-the-art ones in HAR:C is adjusted to yield the highest performances for each method separately.Each of the models aforementioned is used to learn features on a training and testing datasets. To evaluate the relevance of the obtained features, we use them to train and evaluate a standard classifier common to all approaches. We decide to use a soft-margin SVM  with a lIt should be noted that many of the feature extractors in our comparative study require a certain number of hyper-parameters  to be set. The choice of those parameters has been shown to have a high impact on the final classification performances  or more elaborated descriptors such as features related to the frequency-domain based on the Fourier transform of signals . DespiteThe idea behind the codebook approach is based on two consecutive steps, illustrated in w and sliding stride l is firstly applied on each sequence to obtain subsequences . A k-means clustering [N clusters of similar subsequences, with the Euclidean distance used as similarity metric between two subsequences. It can be noted that the outcome of k-means clustering can be dependent on the initialization of the cluster centers, which is performed at random. We therefore run the clustering algorithm 10 times, and select the result for which the sum of the Euclidean distances between subsequences and their assigned cluster centers is minimal. At the end of this step, a codebook consisting of N codewords is obtained.For the codebook construction step, a sliding time window approach with window size ustering  algorithN codeword frequencies yields one to obtain a probabilistic feature representation.For the codebook assignment step, subsequences are firstly extracted from a sequence using the same sliding window approach employed in the codebook construction step. For each subsequence, the most similar codeword is determined. A histogram of the frequencies of all codewords is then built, with the bin corresponding to one codeword being incremented by one each time the codeword was considered as the most similar to one of the subsequences. The histogram is finally normalized so that the sum of the hard assignment” (from now on abbreviated as CBH) since each subsequence is deterministically assigned to one codeword. This approach however might lack flexibility to handle “uncertain” situations where a subsequence is similar to two or more codewords. To address this issue, we therefore use a soft assignment variant (referred to as CBS) which performs a smooth assignment of a subsequence to multiple codewords based on kernel density estimation [sth subsequence  including an input, one or more intermediate (hidden) and an output layer, as shown in Multi-Layer-Perceptron (MLP) is the simplest class of ANN, and involves a hierarchical organisation of neurons in layers. MLPs comprise at least three fully-connected layers  are a class of ANNs featuring directed cycles among the connections between neurons. This architecture makes the output of the neurons dependant on the state of the network at the previous timestamps, allowing them to memorize the information extracted from the past data. This specific behavior enables RNNs to find patterns with long-term dependencies.vanishing or exploding gradient, which refers to the phenomenon happening when derivatives of the error function with respect to the network weights become either very large or close to zero during the training phase [gates, each featuring their own weights, bias, and activation functions. An input, output and forget gates are used to respectively block the input of the cell  and many-to-one (only the output of the last cell is used as input of the next layer). The output of the last layer can be passed to a dense and softmax layers for the classification problem.Similarly to regular neurons, LSTM cells are organized in layers, with the output of each cell being fed to its successive cell in the layer and to the next layer of the network. LSTM layers can be organized following several patterns, including in particular n-dimensional time series with n being the number of neurons of the convolutional layer. This output is then sliced along the time dimension. Each slice, indicated in blue in Thanks to the high modularity of ANN architectures, it is also possible to append LSTM layers to convolutional blocks as depicted in encoder and a decoder, as shown in An autoencoder (AE)  uses a sThe choice of a dataset is a paramount parameter to establish for any experimental setup. To choose the datasets for our comparative study, we took into account the “complexity” of the data , as “solved” datasets would not allow us to find meaningful differences in the performances of the different feature learning approaches. We selected the OPPORTUNITY  and UniMThe OPPORTUNITY dataset  is builtAs it can be seen in  such as ,13): the such as ,13,15,16The OPPORTUNITY dataset contains a certain number of sensors for which small to very large chunks of data are missing, due to data transmission problems of the wireless sensors during the acquisition process. To bypass this problem, we decided to exclude the sensor channels the most affected by this issue. 38 of them were removed, including data from the left and right hand accelerometers, and quaternion information from the inertial measurement units. For the remaining 107 sensor channels, wecomplement the missing values with the last non-missing one.T and S  for our training set, and from the runs ADL 4 and ADL5 for our testing set. This leads to 211,265 and 78,029 frames for the training and testing sets, respectively. Because of the timestamp-level labeling, each frame can usually contain data associated with multiple labels. We opt for a majority labeling, where the label of each frame is determined as the most frequent label among those of the The UniMiB-SHAR dataset (University of Milano Bicocca Smartphone-based HAR)  aggregatt when the magnitude of the signal is higher than g being the gravitational acceleration constant) and lower than 0 at time t-1. The dataset contains 11,771 time windows of size For this dataset, time windows of data and their associated labels are directly provided with a fixed length ion data . A peak This section presents the implementation details of the different feature learning approaches and their comparative results on the OPPORTUNITY dataset.The following sub-sections provide the implementation details of each feature learning approach, and the architectures of the deep-learning models. All the ANNs were coded in Python using the Keras framework  with a TWe use 18hand-crafted features  . Last, w is set to 24 by examining Given a frame, CB extracts a feature from each of  feature . In thisformance ,39, so lIt should be noted that CB encodes each frame into a internal covariate shift, which refers to the changes in the distributions of the inputs of a layer during the training phase, and can cause the input values to move toward the saturation regime of the activation function. This phenomenon can potentially decrease gradient values and slow down the training phase. We obtained our best performances by placing a single batch normalization layer directly after the input of the network, although [We used a MLP with three hidden layers with REctified Linear Units (RELU) activations, taking vectors obtained by flattening frames of data as inputs. We noticed that the addition of a batch normalization layer  significalthough  recommenOur CNN architecture involves three consecutive blocks, each including a convolutional, RELU activation and max-pooling layers. Each convolutional kernel performs a 1D convolution over the time dimension, for each sensor channel independently . Similarly to MLP, adding a batch normalization layer right after the input layer yields significant performance improvements. The model selected in the end comprises in order a batch normalization layer, three blocks of convolutional-RELU-pooling layers, a fully-connected layer and a softmax layer. The hyper-parameters of those layers are provided in T. All cells of the network use the sigmoid function for the gate activations, and hyperbolic tangent for the other activations. Each LSTM cell of the first layer takes as inputs the data from all sensor channels at its corresponding time, and the output of the LSTM cell processing the data at the previous time. The first LSTM layer is organized in a many-to-many manner.The second LSTM layer follows a many-to-one outlineA batch normalization layer is also added at the beginning, and a dense and softmax layers at the end of the network. The values of all hyper-parameters of the model are provided in Based on preliminary experiments, we decide to use a model with two layers. The number of LSTM cells in both layers is equal to the size of the time window Using an architecture similar to the ones in , we stacWe use a simple autoencoder model, with the encoder and the decoder both featuring only one dense layer. This decision stems from our observations that increasing the number of hidden layers did not improve the classification performances, and that using layers with many neurons instead  . We firsS leads to better performances for all different methods, as the variety and amount of data used to train the models increase. The relative performances of the feature extraction approaches also remain unchanged for all the tested numbers of sensor channels. Hybrid models still outperform all other approaches, which confirms the robustness of such deep-learning architectures.As expected, increasing T has a high influence on the features extracted by the different feature learning models. This is especially the case for the deep-learning related methods, for which a variation of the size of the input layer impacts the architecture of the subsequent layers, and causes changes in the field of view of the neurons in convolutional layers, or in the number of LSTM cells of the recurrent layers.The length of the sliding window T, we carried out two more comparative studies using To check whether the trends we observed previously are still relevant regardless of T. The supervised deep-learning methods still achieve the best performances, with hybrid models outperforming the other methods.The trends observed in the reference case  employed on the UniMiB-SHAR data remained unchanged compared to those used on the OPPORTUNITY dataset  presented in The codebook approach with soft-assignments yields the highest accuracy and weighted F1-score at The results obtained on both datasets highlight three points: the superiority of automatically learned features over hand-crafted ones, the stability of hybrid CNN-LSTM architectures which are able to obtain top performing features on both datasets, and the relevance of the codebook approach with soft-assignment to some specific setups.If the bad results obtained by hand-crafted features compared to the other feature learning approaches are not unexpected, the gap in their relative performances displayed on both datasets is more surprising: the difference between their average F1-score and the one of the best performing method is n method  to deterOne interesting observation is the dependence of the performances of deep-learning models on the segmentation phase. Depending on whether the latter is sliding-window-based  or energy-based , the type of data frames varies, thus impacting the type of predominant features for the classification. Features characterizing the shape of the time signals— which are typically obtained by convolution-based networks—perform well in the case of an energy-based segmentation, whereas a sliding window segmentation approach makes the memorization of previous data frames more relevant and the LSTM-based architectures more relevant. It can be noted that the hybrid CNN and LSTM model still ranks among the most efficient methods on UniMiB-SHAR, which highlights the stability of this model, and suggests its good generalization capacity on different datasets.The good performances of CBS on the UniMiB-SHAR dataset is another indicator of the effectiveness of low-level shape-based features in the case of an energy-based segmentation. As we found out that the codebook and deep-learning-based approaches produce different features which yield significantly different results, we expect their fusion to take advantage of this “diversity” and lead to performance improvements . We thusIn this paper, we defined an evaluation framework which fixes all stages of the ARC except the feature extraction step to rigorously compare feature learning methods for sensor-based HAR. We applied this framework on two benchmark datasets, OPPORTUNITY and UniMiB-SHAR, to compare and evaluate the performances of several state-of-the-art methods. The results indicate that a hybrid deep-learning architecture featuring convolutional and LSTM layers is a flexible and robust approach, yielding top performances on both datasets. Changes in hyper-parameters related to the other steps of the ARC, such as the frame size or the number of sensor channels did not change the overall ranking of the feature learning models. The codebook approach also showed to be efficient for an energy-based segmentation case, such as the UniMiB-SHAR dataset. The fusion of features learned by both approaches further improved the results. All research materials used in our studies  are also provided to allow the reproduction of our results, and let other researchers test their own features on the datasets we used and rigorously compare their performances to ours.