Android Library for Recognition of Activities of Daily Living: Implementation Considerations, Challenges, and Solutions

1.1. Background

Mobile devices like smartphones commonly used in daily life can be classified as multi-sensor devices [1], because they host several sensors, e.g., accelerometer, gyroscope, magnetometer, microphone, Global Positioning System (GPS) receiver and camera, that allow the acquisition of different types of data, including some of the physical and physiological data pertaining to the user, thus allowing for the recognition of the Activities of Daily Living (ADL) [2, 3].

The development of Ambient Assisted Living (AAL) systems has many challenges [4-8], such as timely and accurate recognition of ADL [9, 10], designing architectures that are applicable for Enhanced Living Environments [11, 12], providing reliable systems and integration with other systems [13]. ADL recognition is also important to design a personal digital life coach [14]. The development of these systems is specially important to support the autonomy of older users, patients with chronic diseases and users that may have some type of disability [15, 16]. However, these systems may be useful for everyone, including athletes and young users, as the proposed framework can be integrated into a tool for the monitoring and training of lifestyles [14].

1.2. Motivation

The recognition of ADL and the environment where the subject is at a given moment is widely important for the creation of systems aimed at several purposes, such as the personal coaching, the health monitoring, the lifestyle monitoring and others. Nowadays, such recognition is commonly performed by complex and expensive systems with high power processing and memory capabilities [4-6, 11-13], but it could be performed with commonly used equipment based on local processing techniques, in order to provide a feedback related to the ADL performed and environment frequented in almost real-time.

Our motivation is to solve this problem with the development of an Android library that performs the recognition of ADL and the associated environments, using only the data acquired from the sensors available onboard off-the-shelf mobile devices (i.e., smartphones), and local processing. We selected the Android platform for the development of the library, because this is the most used open platform available in the market [17], and the architecture of an Android library for the recognition of ADL and environments has been already proposed in [18-20].

The use of the off-the-shelf mobile devices for this purpose requires the development of lightweight processing techniques. In fact, despite recent developments and improvements, these devices usually have several limitations, including reduced processing capabilities, low capacity batteries and small internal storage space.

However, based on previous studies [21-25], we confirm that the use of the sensors available in off-the-shelf mobile devices may fuel the development of ADL recognition systems, having the potential to improve the quality of life of its users. With this study, we present a real-time mobile application that reliably identifies ADLs, the environment and location, providing feedback to the user. The recognition of the ADL featuring the subject’s movement is based on the data acquired from the motion and/or magnetic sensors (i.e., accelerometer, gyroscope and magnetometer), but the recognition of the environment is based on acoustic data acquired from the microphone. In order to support the better recognition of an ADL when the user is standing still, the fusion of the features extracted from the data acquired from motion and magnetic sensors with the environment previously recognized and the features extracted from the data acquired from the GPS receiver is performed.

1.3. Prior Work

To date, and based on the literature, there are several studies using different subsets of sensors available on smartphones. Yet, studies reporting methods that rely on a larger set of sensors available in off-the-shelf mobile devices for the recognition of ADL and their environments were not found. For the purpose of comparison with this research, we analyzed 36 studies available in the literature that use Artificial Neural Networks (ANN) for the recognition of ADL and environments. The implementation of ANN with mobile devices is widely studied and several frameworks have been developed. In this case, other types of methods have been already used in the literature, including the neural networks and the ensemble learning methods [26], but the ensemble learning methods often include a combination of methods (thus requiring higher computational resources) and can be used with larger datasets than the neural networks methods, which will be the case for the datasets available in the storage of mobile devices [27].

The use of the ANN, with variants, to recognize different types of ADL involving the subject’s movements, is quite well documented in the literature. Several works, as the following discussion shows, addressed this method, providing different accuracy performance on different datasets. There are several studies using only the accelerometer sensor, where the authors of [28] recognized the walking and standing activities using Multilayer Perceptron (MLP) with a reported accuracy around 95%. Bayat et al. [29] implemented the MLP method using features extracted from the accelerometer data for the recognition of running, walking, dancing, going upstairs and going downstairs, reporting an accuracy of 89.72%. In [30], the authors used the accelerometer data and implemented the MLP method for the recognition of going upstairs, going up on an escalator and walking, reporting an average accuracy of 94%. The study of [31] implemented the MLP method for the recognition of several activities, such as walking, jogging, going upstairs, going downstairs, sitting, standing and lying, reporting a maximum accuracy of 94.44%. The standing, walking and running activities reported a minimum accuracy of 89% using the MLP method [32]. The authors of [33] implemented the MLP method for the comparison of the results obtained with the Sliding-Window-based Hidden Markov Model (SW-HMM), reporting an accuracy always higher than 80% in the recognition of walking, standing, running, going downstairs and going upstairs. MLP method was implemented by the authors of [34] for the recognition of standing, sitting, walking, lying and falling activities, reporting an accuracy higher than 90%. Using the same method, in [35], the authors reported that the recognition of the standing, walking, jogging, jumping, going upstairs, going downstairs and sitting activities has an accuracy higher than 90%. In [36], the walking and standing activities reported a recognition accuracy between 75% and 98% using the MLP method. The MLP method was used by the authors of [37] for the recognition of standing, walking, going upstairs, going downstairs and running activities, reporting an accuracy of 92%. Cycling, standing, walking, jogging, driving and running are other activities recognized by the authors of [38] with the MLP method, reporting an accuracy between 57.53% and 97.58%. In [39], the MLP method was used for the recognition of walking, going upstairs, going downstairs, standing, sitting and jogging activities, reporting an accuracy around 97%. In order to identify the activities performed during a football game, the authors of [40] used the MLP method with a reported accuracy higher than 95%. In [41], the walking, going upstairs, jogging, going downstairs, standing and sitting activities were recognized with the MLP method, which reported an accuracy of 97.7%. The study [42] presented the implementation of the MLP method for the recognition of walking, jogging, going upstairs, going downstairs, sitting and standing activities, reported an average accuracy of 82.3%. In [43], the MLP method was used for the recognition of cycling, running, sitting, standing and walking with a minimum reported accuracy of 61%. In addition to the use of the MLP method with the accelerometer data as input, other methods are growing, such as DNN and Probabilistic Neural Networks (PNN) methods. DNN method was used by the authors of [44] for the recognition of walking, running, standing, sitting, lying, going upstairs and going downstairs activities with a reported accuracy between 77% and 99%. PNN method was used by the authors of [45] for the recognition of walking, jogging, cycling, going upstairs and going downstairs activities, reporting results with an average accuracy of 98%.

Following the studies that fuse the data acquired from the accelerometer and the gyroscope sensors, the authors of [46] used the MLP method for the recognition of standing, walking, running, sitting, lying, cycling, driving, cleaning, cooking, taking medication, sweeping, washing hands and watering plants activities, reporting an average accuracy of 65.44%. The MLP method was used in [47] for the recognition of walking, jumping, running, going downstairs and going upstairs, reporting an average accuracy of 84.53%. For the recognition of falling activities, the authors of [48] implemented the MLP method and reported an accuracy of 85.87%. The walking, jogging, sitting, going upstairs and going downstairs were recognized with the use of the MLP method, reporting an average accuracy of 71.25% [49]. In [50], the authors implemented the MLP method for the recognition of movements related to getting up and down from buses, reporting an average accuracy of 86%.

A system that uses Support Vector Machines (SVMs) with tuned parameters and features based on histograms extracted from multiple accelerometers and gyroscopes is proposed in [51]. This is a valid approach because such features are easy to compute. A balanced accuracy over 85% is reported.

An approach based on automatic feature engineering is used in [9] for the recognition of a variety of ADLs, using sensors placed on multiple locations. Depending on the dataset, an accuracy from 90 to 99% is reported. Interestingly, the best accuracy achieved with SVMs, Random Forest and Extra Trees is comparable to the accuracy of much more lightweight approaches like logistic regression and Naïve Bayes. A similar approach was proven to work for jogging detection in [52], even on field conditions when the subject wears accelerometers all the time.

In [53], the accelerometer and gravity sensors’ data were fused and the MLP method used for the recognition of walking, going upstairs, going downstairs, standing and laying activities, reporting an accuracy higher than 90%.

Combining the data acquired from the accelerometer and the GPS receiver, the MLP method handles the recognition of standing, walking, and travelling by car and train, reporting a minimum accuracy of 70% [54].

In [55], the data acquired from the accelerometer, the magnetometer, the gyroscope and the GPS receiver were fused and the MLP method was applied for the recognition of running, walking, sitting, going upstairs, going downstairs and sitting activities, reporting a recognition accuracy higher than 70%.

The acoustic data may be exploited for the recognition of the different environments using ANN. Authors of [56] implemented the MLP method for the recognition of sounds of emergency vehicles, reporting an accuracy around 96.70%. In [57], several sounds have been recognized with the MLP method, such as boll impact, metal impact, wood impact, plastic impact, open/close door, typing, knocking, telephone ringing, grains falling, spray and whistle, reporting an average accuracy in environment recognition of 92.41%. The sounds of sneezing, dog barking, clock ticking, baby crying, crowing rooster, raining, sea waves, fire crackling, helicopter and chainsaw were recognized in [58] using the MLP method and reporting an accuracy around 94.5%. The authors of [59] recognized several sounds, including alarms, birds, clapping, dogs, motorcycles, raining, sea waves and wind, based on the application of the MLP method, and reported an accuracy around 54%. Other ANN methods have been used, such as Feedforward neural networks (FNN), Recurrent Neural Networks (RNN) and DNN methods. The FNN were implemented by the authors of [60] for the recognition of emergency vehicles, car horns, and normal street sounds, reporting an accuracy between 80% and 100%. The authors of [61] implemented DNN and RNN methods for the recognition of the sounds related to baby crying and smoking alarm, reporting an accuracy around 90%. The DNN methods was also used for the recognition of different sounds, including laughing, singing, crying, arguing, sighing, voice and music, reporting accuracies around 80% [62, 63].

The acoustic data may also be used to help in the recognition of some ADL; the authors of [64] recognized the falling events with the MLP method, reporting an accuracy around 93%.

A summary of the literature overview discussed above is provided in Tables 1 to 5, where details regarding the number of ADLs analyzed by each study, and the accuracy obtained on average and on the single type of ADL considered are presented. However, the accuracy values reported by the studies analyzed are not directly comparable as they were obtained from different datasets.

Taking into account the set of studies we selected and analyzed from the literature, as listed in Tables 1 to 5 and in the bibliography section of this work, we can conclude that the majority of the studies (51.43%) uses only the accelerometer for the recognition of ADL, and only 9 studies (25.71%) used acoustic data for the recognition of the environments. Nevertheless, 5 studies (14.29%) present methods that fuse the data acquired from the accelerometer and gyroscope sensors. Other combinations are investigated in the literature, such as the fusion of the data acquired from the accelerometer and GPS receiver (2.86%), the fusion of the data acquired from the accelerometer and the gravity sensors (2.86%), and the fusion of the data acquired from the accelerometer, the magnetometer, the gyroscope and the GPS receiver (2.86%).

The number of ADL recognized by each study varies between 1 and 13 but in 96% of the studies that number varies between 1 and 7. In half of that studies, the number of ADL recognized is up to 5 included. In case of the number of environments recognized by each study, they vary between 1 and 11, but its dispersion is enormous. In 62.5% of that studies, the number of environments recognized is up to 5, and the rest (37.5%) are 10 and 11.

Tables 1 to 5 present the mapping between the study analyses and the ADL and environments recognized, including the accuracies reported for further comparison with our study. The accuracies presented are related to each ADL and environment recognition.

Finally, Tables 6 to 7 summarize the average, standard deviation and the coefficient of variation of the standard deviation of the accuracy for each ADL and environment recognized in the previous work analyzed. The low coefficient of variation of standard deviation values (<30%) permit conclude that both average accuracy of ADL and environments recognized are representative.

1.4. Purpose of this Study

Based on our previous work [65-68], the purpose of this study consists in the development of an Android library that, depending on the number and type of sensors available onboard the mobile device, implements the framework previously designed for the recognition of the ADL and their environments exploiting the major number of sensors available in the mobile device used. The implementation should use lightweight methods in a mobile application in order to give a rapid feedback to the user with the data processing methods implemented locally on the mobile device.

The previous work explored the use of different implementations of ANN, such as MLP with Backpropagation, FNN method with Backpropagation and DNN method with normalized and non-normalized data. The framework implemented in this research is composed of several stages, these being: data acquisition, data processing, data fusion and classification methods. The data acquisition and processing depends on the types of sensors, where for the accelerometer, gyroscope and magnetometer data a low-pass filter is applied; for the acoustic data the Fast Fourier Transform (FFT) for the extraction of the relevant frequencies of the audio signal is applied; and for the location data no filters are applied as the raw data acquired is used for the measurement of the distance travelled [69]. Following the application of the low-pass filter and the FFT to clean the data acquired, some features related to the sensors’ signals were extracted and fused. Finally, the classification methods were applied for the recognition of ADL and environments.

The recognition of ADL and their environments are separated into three stages. These are the recognition of common ADL (i.e., walking, going upstairs, going downstairs, running, standing and watching TV), the recognition of the environments (i.e., bar, classroom, bedroom, gym, hall, kitchen, library, street and living room), and the recognition of ADL without body motion (i.e., sleeping, watching TV and driving). Based on the outcomes of previous works, firstly, for the recognition of the common ADL the method that reported better results was DNN method with normalized data. Secondly, for the recognition of environments, the method that reported better results was FNN method with non-normalized data, but, when compared with the previous results, this method reports a low performance and accuracy during the implementation of the Android library, where the results obtained are better with DNN method with normalized data. Finally, for the recognition of ADL without body motion, the method that reported better accuracy was also DNN method with normalized data. The data used in this study is available in the ALLab MediaWiki [70].

1.5. Structure of this Study

This section presents our motivation and related work for the development of an Android library for the recognition of ADL and their environments. The following sections present the methods used and results obtained with the developed Android library, including their discussion and comparison with research studies available in the literature with a similar purpose.

2.1. Study Design and Participants

Off-the-shelf mobile devices are equipped with several sensors, e.g., accelerometer, gyroscope, magnetometer, microphone and GPS receiver, which allow the acquisition of several physical and physiological parameters.

This study is focused on the development of an Android library [71, 72] for the processing of the sensors data’ and recognition of ADL and their environments. The mobile device used for the tests of the Android library was a BQ Aquarius device [73], but the sampling frequency of the sensors’ data acquisition is difficult to measure, because these devices have several constraints, mostly because as the system operates in a time-sharing manner, it is impossible to assure a steady microprocessor attention to the sensor related processes.

Subjects (15 males and 10 females) aged between 16 and 60 years old were selected for the data acquisition with a mobile application that uses the Android library developed. Independently of the gender, we have selected people with distinct lifestyles, where 10 subjects self-assessed their living style as mainly active and the other 15 subjects self-assessed their living style as mainly sedentary. During the use of the mobile application, the mobile device is performing other tasks, such as receiving calls, using the Internet connection and using other mobile applications. However, the major part of the time, the mobile device is in the front pocket of the user’s pants for the correct data acquisition.

The developed mobile application acquired the sensors’ data as a background service, acquiring and processing slots of 5 seconds of data. With only 5 seconds of data, the ADL and environments will be recognized. The data acquisition is performed every 5 minutes, and 2000 captures for each ADL (i.e., walking, running, going upstairs, standing/watching TV, going downstairs, driving and sleeping) and environment (i.e., bedroom, bar, classroom, gym, kitchen, living room, hall, street and library), which corresponds to 10000 seconds (approximately, 2.7 hours) for each ADL and environment. During the performance of the experiments, the user should label the ADL and environment acquired because these data were used to train and test the methods for the classification, which are implemented in the Android library.

2.2. Description of the Framework

Following the previous work that explains the architecture and the method with best results in the recognition of ADL and their environments [18-20, 65-68], the framework implemented is a collection of the methods that, according to the literature, provide the best performance on recognition of ADL and environments, subjected to the available sensors in the mobile device where the framework is executed. Thus, its development is separated into several modules, such as data acquisition, data processing, data fusion and classification methods.

Data acquisition methods are not provided by the Android library, performing only the data processing, data fusion and classification methods. Data acquisition of the accelerometer, gyroscope, magnetometer, microphone and GPS data should be performed with the standard methods of Android platform, using a mobile application. For the remaining modules, the Android library should be a function of the number of sensors available in the mobile device.

Data processing methods implemented in the Android library consist of the data cleaning and feature extraction methods. The data cleaning method used depends on the type of sensors, i.e., for accelerometer, gyroscope and magnetometer data, the low pass filter is applied, and, for the acoustic data, the FFT is applied for the extraction of the relevant frequencies. The feature extraction also depends on the type of sensors, i.e., for accelerometer, gyroscope and magnetometer data, the features extracted are the five greatest distances between the maximum peaks combined with the Average, Standard Deviation, Variance and Median of the maximum peaks, and the Standard Deviation, Average, Maximum, Minimum, Variance and Median of the raw signal. For the acoustic data, the features extracted are the 26 Mel-Frequency Cepstral Coefficients (MFCC) combined with the Standard Deviation, Average, Maximum, Minimum, Variance and Median of the raw signal. Finally, for the location data, the feature extracted is the distance travelled.

After the data fusion of the features extracted from the sensors’ data, the normalization and classification methods are applied in three stages:

• Stage 1:
  • Sensors: accelerometer, gyroscope and/or magnetometer;
  • Purpose: Recognition of walking, standing, running, going upstairs and going downstairs activities using DNN method with normalized data.
• Stage 2:
  • Sensors: Microphone;
  • Purpose: Recognition of bar, bedroom, classroom, gym, hall, kitchen, street, library and watching TV/living room environments using FNN method with non-normalized data and/or DNN method with normalized data.
• Stage 3:
  • Sensors: accelerometer, gyroscope, magnetometer and/or GPS receiver;
  • Purpose: Recognition of watching TV, sleeping and driving activities using DNN method with normalized data.

Related to the stage 2, the previous work [65] recommends the use of FNN method with non-normalized data. However, as it will be presented in the following sections, we verified that the accuracies could be improved by combining different methodologies:

1. Use of FNN method with non-normalized data, and, when the results are not clearly identified, the use DNN method with normalized data for these cases may perform a better recognition of the environments (Fig. 1A);
2. Use only the DNN method with normalized data (Fig. 1B).

The FNN method with non-normalized data was implemented, trained and tested with the Encog framework [74] in a Java desktop application, creating a neural network for further application in the mobile application. The DNN methods with normalized data were implemented, trained and tested with Deeplearning 4j framework [75] in a Java desktop application, creating a neural network for further application in the mobile application. The train of the neural networks needs a significant processing effort, that is performed previously on a desktop computer, using only the method developed in the mobile application. There are several parameters used for the creation and training of the neural network, including the activation function, and the learning rate and momentum values used, which are presented in Table 8. These configurations may influence the speed of the training stage as well as the results obtained with the trained network file.

2.3. Test of the Method

We created a set of methods to test the correct identification of the ADL and environments using the Android library we developed. For this purpose, we acquired the sensors’ signal with a mobile device related to around 2.7 hours for each ADL and environments, correspondent to 2000 records with a length of 5 seconds, which are labelled in the mobile application in order to allow the measurement of the accuracy of the Android library developed. We implemented unit tests to evaluate the previously acquired records, reporting the ADL or environment recognized with each record in order to measure the overall recognition accuracy (percentage) of the Android library as well as the recognition accuracy (percentage) of each ADL and environment with the Android library.

3.1. Prototype

The main result of this study consists of the development of an Android library that implemented the framework for the recognition of ADL and their environments. In addition, a mobile application, with a Graphic User Interface presented in Fig. (2), that embeds the Android library developed, was implemented for its validation.

The main functionality of the mobile application developed consists in the presentation of the current or the last ADL, environment or location recognized (Fig. 2A and B). Based on the calendar, the mobile application has a module to manage the ADL and environments planned (Fig. 2C and D), Finally, the mobile application includes a module to show the list of the ADL and environments recognized by each calendar day (Fig. 2F), including the highlight of the respective calendar days that included the ADL and environments recognized (Fig. 2E).

In the following sections, the validation and results of the Android library embedded in the mobile application will be shown, proving the reliability of the framework for the recognition of ADL and their environments.

3.2. Requirements

This study consists of the use of the sensors available in consumer mobile devices. Due to the different manufacturers in the market, these devices have a different number of sensors, and the Android library developed should be flexible and adaptable to the number of sensors effectively available, allowing their use with all devices available on the market.

To use our Android library, the mobile device should have a minimum number of sensors (i.e., accelerometer and microphone), but, depending on the number of sensors included in each mobile, the final accuracy of the recognition of ADL and their environments will be different. Thus, if the mobile device includes the GPS receiver, the mobile application will increase the number of ADL correctly recognized. And, if the mobile device has a network connection, the user’s location will be shown in the mobile application.

In conclusion, depending on the number of sensors available in the mobile device, the Android library is able to recognize between 5 and 7 ADL, and 9 environments, as shown in the next section.

3.3. Validation

Once the framework for the recognition of ADL and environments in the Android library was implemented, we tested both the different stages of the framework independently and the overall execution of the framework. For these tests, we took into account the different combinations of sensors possibly available in the mobile devices. These tests were performed with the dataset previously acquired and available at [70].

Based on the implementation of stage 1 for the recognition of common ADL (i.e., standing, walking, running, going upstairs and going downstairs) using the DNN method with normalized data, we achieved an average accuracy of 86.39% with the accelerometer sensor (Table 9), 86.49% with the accelerometer and magnetometer sensors (Table 10), and 89.15% with the accelerometer, magnetometer and gyroscope sensors (Table 11).

During the implementation of stage 2 for the recognition of environments (i.e., bedroom, bar, classroom, gym, kitchen, watching TV/living room, hall, street and library), we verified that the FNN method with Backpropagation used in the previous study [65], reported confused results. In fact, using the FNN method with Backpropagation, the framework reports an average accuracy of 22.68% in the recognition of the environments, recognizing only 4 out of 9 environments (Table 12). As the results were not good, we created a new method in the framework that first tries the recognition of the environments using an FNN method with Backpropagation, and, when the results are not clearly identified, automatically applies the DNN method with normalized data. This way, we obtained an average accuracy of 30.27% with a slowing down performance of the mobile device, recognizing 8 out of 9 environments (Table 13). Finally, we implemented the DNN method only with normalized data for the recognition of the environments, increasing the performance of the Android library, and reporting an average accuracy of 45.68% recognizing all the environments (Table 14). In the next analysis related to the recognition of the environments, the results obtained with the DNN only method will be taken into account.

Based on the implementation of stage 3 for the recognition of ADL without body motion (i.e., watching TV, sleeping and driving), the results obtained are always 100%, but these results depend on the results obtained during the previous stages of the framework for the recognition of ADL and their environments, because the outcomes depend on the correct recognition of the environment (stage 2), the correct recognition of the ADL as standing (stage 1) and the correct extraction of the GPS features.

Finally, we tested the overall recognition of the ADL (i.e., walking, running, going upstairs, going downstairs, watching TV, sleeping and driving) with the sequential execution of the three stages using only the DNN method with normalized data, reporting an average accuracy of 58.02% with the use of the accelerometer and the environment recognized (Table 15); 58.34% with the use of the accelerometer, the magnetometer and the environment recognized (Table 16); 60.04% with the use of the accelerometer, the magnetometer, the gyroscope and the environment recognized (Table 17); 67.05% with the use of the accelerometer, the GPS receiver and the environment recognized (Table 18); 67.26% with the use of the accelerometer, the magnetometer, the GPS receiver and the environment recognized (Table 19), and, finally, 69.43% with the use of the accelerometer, the magnetometer, the gyroscope, the GPS receiver and the environment recognized (Table 20).

4.1. Main Findings

Based on the partial results obtained in the previous work [65-68], we present a summary of the accuracies obtained in each stage of the framework for the recognition of ADL and their environments (Tables 21 to 23), for further comparison with the results obtained with the Android library developed for this study (Tables 24 to 26), which combines the different stages all together.

Following the results obtained in the stage 1, related to the recognition of the common ADL, with the implementation of the DNN method with normalized data, we verified that, with the Android library developed, the ADL are recognized with more accuracy (+0.50%) with the accelerometer sensor, with the same accuracy with the joint accelerometer and magnetometer sensors, and with less accuracy (-0.36%) with the accelerometer, magnetometer and gyroscope sensors than the accuracies obtained in the previous work [65-68]. Table 27 shows the values of the accuracy (A), the specificity (S), the recall (R) and the precision (P) of the recognition of each ADL as well as the average of these parameters in the Android library developed, verifying that the ADL with highest recognition accuracy is the standing with a precision of 97.11% and a recall of 99.25%, increasing the results in the framework, because it influences the results of stage 3.

Following the results obtained in stage 2, related to the recognition of the environments, we tried to use the same method implemented in the previous works [65-68], which is the FNN method with Backpropagation and non-normalized data, but instead of the 86.50% of the recognition accuracy, the Android library reported an accuracy of 22.68% and only 4 environments were recognized. However, due to the capabilities of the mobile devices, the recognition obtained was very low, and we tried to combine the use of the FNN method with Backpropagation and non-normalized data with the use of the DNN method with normalized data, which proves that the accuracy obtained increases to 30.27%, recognizing 8 of the 9 proposed environments. As we verified that the use of the DNN method with normalized data increases the recognition accuracy of the environments, we tried to implement only the DNN method with normalized data, reporting an accuracy of 45.68%, recognizing all of 9 environments with better performance, because the execution of the DNN method requires less resources than the execution of the FNN method with Backpropagation. Table 28 shows the values of the accuracy (A), the specificity (S), the recall (R) and the precision (P) of the recognition of each environment as well as the average of these parameters in the Android library developed, verifying that the results with the highest precision were achieved in the street and the bedroom environments and the highest recall value in the recognition of the bedroom environment.

Following the results obtained in stage 3, related to the recognition of the ADL without body motion, with Android library developed, the ADL without body motion is recognized with 100% of recognition accuracy, which is the same accuracy reported by the previous works [65-68]. Tables 29 and 30 show the values of the accuracy (A), the specificity (S), the recall (R) and the precision (P) of the recognition of each ADL without body motion as well as the average of these parameters in the Android library developed, verifying that the values of the accuracy, recall, specificity and precision are always 100%.

Following the overall results obtained by the framework in the previous works [65-68], for the measurement of the overall accuracy, we took in account that when the ADL was recognized as an activity without motion, it was always correctly classified. Thus, using the same dataset for the tests, in the Android library, we tested the flow of the framework for the recognition of ADL, reporting accuracies between 58.02% and 86.49% with the accelerometer sensor, between 58.34% and 86.49% with the accelerometer and magnetometer sensors and between 60.04% and 89.15%, depending on the number of ADL recognized that varies between 5 and 7 ADL. Tables 31 and 32 show the values of the accuracy (A), the specificity (S), the recall (R) and the precision (P) of the recognition of each ADL as well as the average of these parameters in the Android library developed, verifying in general the results obtained reported highest precision and specificity, where the lower accuracies were reported for the going upstairs and downstairs activities.

One of the possible causes of the variability of the results between the previous work and the Android library developed may be the restrictions in hardware and software verified in the mobile devices, that will be analyzed in the next section.

4.2. Limitations

Due to the software and hardware limitations of the mobile devices, their use for the recognition of ADL and environments has several challenges, which are evidenced in several studies [76-81]. These are:

• The use of several types of connection (e.g., Wi-Fi and Bluetooth) decreases the usable battery time;
• During the processing of the sensors’ data, the performance of the mobile device may decrease;
• Due to the execution of several tasks at the same time and the reduced resources of these devices, the frequency of the data acquisition is not constant over the time;
• The placement of the mobile device is one of the most important problems because it is very important for the acquisition of valid data, where there are no studies providing the absolute optimal position that can be good for any type of recognition one wants to perform because it depends on the purpose of the study;
• If the processing of the sensors’ data is performed at a remote server, a permanent data connection should be available; this will also contribute to further decrease the usable battery time;
• The number of sensors available in the consumer mobile devices is different by hardware version and manufacturers, but the developed methods should be a function of the number of sensors available on these devices; moreover, different manufacturers may report different error margins and even different output value ranges;
• Regarding the use of multiple devices, the security, privacy and resilience of the data exchanged over the different types of network connections should be taken into account for the correct use of the mobile systems.

For the minimization of the effects of the aforementioned limitations, several strategies are proposed in our study, these are:

• These research studies may only use the sensors available in consumer mobile devices, reducing the constraints with the connection to external sensors and the battery lifetime;
• The use of lightweight methods may reduce the problems with the reduced power processing of these devices;
• Another possible solution is to enable and disable the acquisition of the sensors’ data over the time because it is not necessary to have a continuous real-time sensor data acquisition in order to recognize the ADL and their environments;
• The creation of methods adapted to the different sets of sensors available in the consumer mobile devices increases the availability of the methods for a large group of people;
• The use of local data processing reduces the time to give feedback to the user, but it also reduces the problems with the security and privacy of the personal data, despite increasing the computation burden of the device.

4.3. Comparison with Prior Work

Based on the previous research studies available in the literature, Table 33 presents the comparison between the minimum, maximum and average of the accuracy achieved and the results reported by the Android library developed. The results that are comparable are highlighted in Table 33, verifying that the highest improvement was verified in the running activity, when we compared our results whit the average accuracy achieved in prior work. For that activity, the minimum and the maximum accuracy founded are 68,70% and 100% respectively, returned a positive (27,15%) and negative improvement (-4.15%).

Following the comparable results, presented in Fig. (3), we verified that our results are always higher than the average of the results obtained in the previous research studies in the literature.

However, the significance of the improvements (see OR-Avg in Table 33) was evaluated with the Student’s t-test for a sample. As presented in Table 34, the normality assumption was achieved, and it was assessed by the Shapiro-Wilk test (T_SW(8)) = 0.889; p_value=0.228). SPSS Statistics Software (v. 24; IBM SPSS, Chicago, IL) [82] was used to perform the statistical tests. The improvements for which the p-value of the test is lower than 0.05 are considered statistically significant, at 95% level of confidence.

The results of the Student’s t-test for the sample used are presented in Table 35, reporting a value of the average of improvements equals to 2.93%. According to the Student’s t-test, this value is statistically significant (t(7)=3.062; p_value =0.018).

In addition, the same procedure was done for analyze the improvements obtained when compared our results, whit the minimum and maximum result founded in the literature (see OR-Min and OR-Max in Table 33, respectively).

In Table 36, the normality assumption was achieved (Shapiro-Wilk test for OR-Min and OR-Max, respectively: (T_SW(8) = 0.914; p_value = 0.381) and (T_SW(8) = 0.968; p_value=0.880)).

Based in Table 37 the results of Student’s test show, in one hand that our results and the minimum accuracy found in the literature are statistically different (t(7) = 3.563; p_value = 0.009), and they are higher (average of the difference between OR and Min is equal 21.0775%). In the other hand, when comparing our results and the maximum achieved in literature, we can accept the negative average of the difference between them (average of the difference between OR and Max is equal -6.2825%) is equal zero (t(7) = -2.076; p_value = 0.077).

Finally, based on the accuracies, presented in the Table 33, and the number of ADL/Environments recognized in the literature, presented in the Table 38, we can conclude that our method is better than other available in the literature review, because our improvement is reliable, because we recognize a major number of ADL/Environments than the other previous works and we obtained statistically significant results.

In addition, we performed the comparison between out number of ADL/Environments recognized with the mean number of ADL/Environments recognized in the literature, evaluating the significance of our number of ADL/Environments recognized with the Student’s t-test for a sample. As presented in Table 39, the normality assumption was achieved, and it was assessed by the Shapiro-Wilk test (T_SW(8)) = 0.889; p_value=0.228). SPSS Statistics Software (v. 24; IBM SPSS, Chicago, IL) [82] was used to perform the statistical tests. The improvements for which the p-value of the test is lower than 0.05 are considered statistically significant, at 95% level of confidence.

The results of the Student’s t-test for the sample used are presented in Table 40, reporting a value of the number of ADL/Environments recognized equals to 16. According to the Student’s t-test, this value is statistically significant (t(8)=0.339; p_value =0.000).

The development of the library has several challenges, including the limited battery lifetime and the reduced processing power of the mobile devices [47]. However, the acquisition of 5 seconds of raw data every 5 minutes using the oldest devices tested has showed a minimum of 16 hours of battery life with a normal usage, and as currently the mobile devices required a daily recharge, it is possible to consider that the acquisition of the sensors’ data with this method can be implemented. The performance of the mobile device is only strictly affected during a small time every 5 minutes, and the effects in the performance are not significant. It will be probably sufficient for the characterization of lifestyles, but it may miss some important events that may not be identified, including falling activities. The development of a method that implements a more significant sampling strategy without decreasing the performance and the availability of resources at the mobile device while identifying a wider number of ADLs requires additional research.

The data processing is executed after the data acquisition for the preparation of the data for further classification in the last stage. The data processing will calculate the features from the raw data, and after that, if the minimum required data is available (i.e., the accelerometer data is correctly acquired), the calculated features will be sent to the classification module, where the verification of the ADL executed is performed and, in case of a similar pattern is not already trained in ANN created for the development of the method in [65-68], the data will be discarded. Nevertheless, the data is only stored until the classification, after which it will be automatically discarded, avoiding problems related to the storage capacity of these devices.