# first we define relevant directories
import sys
# in case PySeqLab package is not installed,
# we can download the package repository from https://bitbucket.org/A_2/pyseqlab
# and then we add the location of the repository to the python system path
# location of the PySeqLab repository on disk -- INSERT location or discard if PySeqLab package is already installed
pyseqlab_package_dir = ""
sys.path.insert(0, pyseqlab_package_dir)
import os
# project directory
project_dir = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
# src directory of this project
src_dir = os.path.join(project_dir, 'src')
sys.path.insert(0, src_dir)
# get the tutorials dir
tutorials_dir = os.path.join(project_dir, 'tutorials')
# to use for customizing the display/format of the cells
from IPython.core.display import HTML
with open(os.path.join(tutorials_dir, 'pseqlab_base.css')) as f:
css = "".join(f.readlines())
HTML(css)
1. Objectives and goals¶
In this tutorial, we will learn about:
- the process/workflow for building classifiers to identify (1) modes of locomotion and (2) gestures movements (i.e. based on CRFs model formalism)
- training the built models and evaluating their performance
- reviving the trained models and decoding new sequences (i.e. test sequences that were not used for training)
1.1 Problem statement¶
The Opportunity challenge (Chavarriaga et al. 2013) is based on a subset of the Opportunity activity recognition dataset comprising of recordings of activity of daily living (ADL) of 12 participants collected in a sensor rich environment (Chavarriaga et al. 2013; Roggen et al. 2010). The challenge is based on recordings of 4 participants who performed 6 different sessions/runs: five daily living sessions, following a loosely defined scenario/situations (see List 1 (Chavarriaga et al. 2013)) and one drill run session in which the participants performed 20 repetitions of predefined sequences (List 2 (Chavarriaga et al. 2013)).
Given the sensor recordings, the objective of the challenge is to build models that can recognize and classify the modes of locomotion (Task A) and gestures movements (Task B). Task A consists of predicting 5 labels/classes and a Null state. Task B consists of predicting 18 labels/classes and a Null state.
List 1. Activity of daily living sequence (Chavarriaga et al. 2013)
ADL run consists of the following activity sequence:
- Start: lying on the deckchair, get up
- Groom: move in the room, check that all the objects are in the right places in the drawers and on shelves
- Relax: go outside and have a walk around the building
- Prepare coffee: prepare a coffee with milk and sugar using the coffee machine
- Drink coffee: take coffee sips, act naturally in the environment
- Prepare sandwich: include bread, cheese and salami, using the bread cutter and various knives and plates
- Eat sandwich
- Cleanup: put objects used to the original place or dish washer, cleanup the table
- Break: lie on the deckchair
List 2. Drill session sequence (Chavarriaga et al. 2013)
Drill run consists of the following sequence:
- Open and close the fridge
- Open and close the dishwasher
- Open and close 3 drawers (at different heights)
- Open and close door 1
- Open and close door 2
- Turn on and off the lights
- Clean table
- Drink (standing)
- Drink (sitting)
Reminder: To work with this tutorial interactively, we need to clone the activity recognition repository from bitbucket to our disk locally. Then, navigate to [cloned_package_dir]/tutorials where [cloned_package_dir] is the path to the cloned package folder.
NB: PySeqLab should be already installed or included in the python system path before we proceed.
2. Training classifiers¶
The src directory in the cloned repository includes three main modules:
train_activityrecognizer_workflow.pysensor_attribute_extractor.pyprocessing_sensor_readings.py
As a prerequisite, refer to the tutorials describing in detail the model building and training process using PySeqLab package.
2.1 Dataset¶
Originally, the dataset (OpportunityUCIDataset) should be downloaded from the UCI public repository and placed under dataset folder in the project's repository(see the printed tree path below):
├── dataset
│── OpportunityUCIDataset
│ ├── README
│ ├── dataset
│ │ ├── S1-ADL1.dat
│ │ ├── S1-ADL2.dat
│ │ ├── S1-ADL3.dat
│ │ ├── S1-ADL4.dat
│ │ ├── S1-ADL5.dat
│ │ ├── S1-Drill.dat
│ │ ├── S2-ADL1.dat
│ │ ├── S2-ADL2.dat
│ │ ├── S2-ADL3.dat
│ │ ├── S2-ADL4.dat
│ │ ├── S2-ADL5.dat
│ │ ├── S2-Drill.dat
│ │ ├── S3-ADL1.dat
│ │ ├── S3-ADL2.dat
│ │ ├── S3-ADL3.dat
│ │ ├── S3-ADL4.dat
│ │ ├── S3-ADL5.dat
│ │ ├── S3-Drill.dat
│ │ ├── S4-ADL1.dat
│ │ ├── S4-ADL2.dat
│ │ ├── S4-ADL3.dat
│ │ ├── S4-ADL4.dat
│ │ ├── S4-ADL5.dat
│ │ ├── S4-Drill.dat
│ │ ├── column_names.txt
│ │ ├── label_legend.txt
│ ├── doc
│ │ ├── ActivityRecognitionBaselines.pdf
│ │ ├── Chavarriaga - The Opportunity challenge-A benchmark database for on-body sensor-based activity recognition (PRL, 2013).pdf
│ │ ├── OPPORTUNITY_D5.1.pdf
│ │ ├── Roggen - Collecting complex activity datasets in highly rich networked sensor environments (INSS, 2010).pdf
│ │ ├── dataset_statistics.pdf
│ │ ├── documentation.html
│ │ ├── img
│ │ │ ├── Fig1Cl.png
│ │ │ ├── Fig1Cr.png
│ │ │ ├── Reed_Switch_Configuration.png
│ │ │ ├── arena3.png
│ │ │ ├── bt_obj1.jpg
│ │ │ ├── label.png
│ │ │ ├── logos-opportunity-final_50p.png
│ │ │ ├── logos-opportunity-final_50p_challenge.png
│ │ │ ├── motion_jacket_complete.jpg
│ │ │ ├── objects.JPG
│ │ │ ├── reed.jpg
│ │ │ ├── usb_sensor.jpg
│ │ ├── locomotion_instances.tex
│ │ ├── mlgesture_instances.tex
│ ├── scripts
│ │ ├── DataExplorer
│ │ │ ├── label_panorama.fig
│ │ │ ├── label_panorama.m
│ │ │ ├── label_panorama_2009.fig
│ │ │ ├── label_panorama_2009.m
│ │ │ ├── readme.txt
│ │ │ ├── root.txt
│ │ │ ├── signal_scope_Callback.m
│ │ │ ├── signal_scroll_Callback.m
│ │ │ ├── underscore_clean.m
│ │ ├── benchmark
│ │ │ ├── RunBenchmarking.m
│ │ │ ├── classifiers
│ │ │ │ ├── GausianClassify.m
│ │ │ │ ├── classifyAndReject.m
│ │ │ │ ├── knn.m
│ │ │ │ ├── nccClassify.m
│ │ │ ├── data
│ │ │ ├── features
│ │ │ │ ├── expandingLabels.m
│ │ │ │ ├── featureExtraction.m
│ │ │ │ ├── featureReduction.m
│ │ │ │ ├── missingValueHandler.m
│ │ │ │ ├── movtimavg.m
│ │ │ │ ├── windowingLabels.m
│ │ │ ├── measures
│ │ │ │ ├── AreaUnderROC.m
│ │ │ │ ├── clsAccuracy.m
│ │ │ │ ├── measures.m
│ │ │ │ ├── rocCurves.m
│ │ │ │ ├── separate.m
│ │ │ │ ├── ward
│ │ │ │ │ ├── README
│ │ │ │ │ ├── a-categorisation-of-performance-errors-in-continuous-context-recognition-ward-2005.pdf
│ │ │ │ │ ├── mset.m
│ │ │ │ │ ├── mset_segments.m
│ │ │ │ │ ├── plot_mset_errors.m
│ │ │ │ │ ├── read_seg_info.m
│ │ │ │ │ ├── test_mset_plotting.m
│ │ │ │ │ ├── wardbars.m
│ │ │ │ │ ├── write_seg_info.m
│ │ │ │ ├── wardbars.m
│ │ │ ├── parameters.m
│ │ │ ├── prepareData.m
│ │ │ ├── readme.txt
│ │ │ ├── res2mat.m
│ │ │ ├── tanalyze.m
│ │ │ ├── tarrange.m
Then we process the data files by running run_data_generation_workflow() function that is found under processing_sensor_readings.py module. The result will be two new generated folders; one for the locomotion task (i.e. task A) and the second for gestures task (i.e. task B). Both folders will include the processed training and test files.
The tree path will look like this:
├── dataset
├── gestures
│ ├── sensor_test_wsize15_stepsize8_yagglast_gestures.pkl
│ ├── sensor_train_wsize15_stepsize8_yagglast_gestures.pkl
│ ├── test.txt
│ ├── test_discretized.txt
│ ├── test_discretized_iob.txt
│ ├── train.txt
│ ├── train_discretized.txt
│ ├── train_discretized_iob.txt
├── locomotion
│ ├── sensor_test_wsize15_stepsize8_yagglast_locomotion.pkl
│ ├── sensor_train_wsize15_stepsize8_yagglast_locomotion.pkl
│ ├── test.txt
│ ├── test_discretized.txt
│ ├── test_discretized_iob.txt
│ ├── train.txt
│ ├── train_discretized.txt
│ ├── train_discretized_iob.txt
We will be using the training and test files (i.e. train_discretized_iob.txt and test_discretized_iob.txt) in locomotion folder and the gestures folder to build our first classifier (task A) and the second classifier (task B) respectively.
Briefly, our data processing pipeline performs the following:
For each task, it aggregates all the original training files representing the sensor recordings by considering each training file to be representing one sequence. Given that we have 14 training files, we will end up with 14 training sequences.
Then we compress the sequences to generate segments using a sliding window of 500 ms (i.e. 15 measurements) and step size of 250 ms (8 measurements). The label of each generated segment was the label of the last measurement in the segment. Moreover, we computed the mean value of each sensor channel in the segment that was later discretized using MDLPC method (Fayyad et al. 1993) implemented in (Raymond, 2013). In the last step, we encode the labels of the generated segments using BIO/IOB format. In this format, each entity class will have two labels: B-class (beginning of entity class) and I-class (within/continuation of entity class). Hence, the total number of classes will be increased to 9 for task A and 33 for task B.
from processing_sensor_readings import *
run_data_generation_workflow()
2.2 Attributes and features extraction¶
We start by defining our attribute extractor that will be used to generate attributes from the parsed sequences. Our attribute extractor SensorAttributeExtractorCateg is subclass of GenericAttributeExtractor class implemented in sensor_attribute_extractor.py module. It defines attributes based on the discretized values of each sensor channel. Below is an example of the attributes extracted using our SensorAttributeExtractorCateg class from a sequence in our training file.
After defining our attribute extractor, we define the feature templates that are used by the feature extractors to generate features. Feature templates and feature extraction are described in detail in this tutorial.
In the train_activityrecognizer_workflow.py module, we define our feature templates using template_config() function.
def template_config():
template_generator = TemplateGenerator()
templateXY = {}
# generating template for tracks
for track_attr_name in track_attr_names:
template_generator.generate_template_XY(track_attr_name,
('1-gram:2-gram', range(-3,4)),
'1-state',
templateXY)
template_generator.generate_template_XY(track_attr_name,
('1-gram', range(0,1)),
'2-states:3-states',
templateXY)
templateY = {'Y':()}
return(templateXY, templateY)
The defined templates include all values of the 113 sensor channel (i.e. s_0:112). The notation s_0:112 represents the recordings of all 113 sensor channels.
For the sensor channels tracks (i.e. s_0:112):
-
- We define a window of size 7 centered at each position in the sequence. We pass through the sequence from left to right, where at each position, we construct a window of size 7 (a window that includes attributes at three previous positions, current position, and three forward/future positions)
- We extract unigrams and bigrams (i.e.
1-gram:2-grams) in the specified window - We join these attributes with (1) the current state (i.e.
Y labels)
-
- We define another window of size 1 centered at each position in the sequence. We pass through the sequence from left to right, where at each position, we construct a window of size 1 (i.e. a window that includes the current attribute).
- We join these attributes using (1) the previous and current state, and (2) the two previous states and the current state (i.e.
Y labels)
from sensor_attribute_extractor import *
example()
2.3 Models and optimization options¶
In the train_activityrecognizer_workflow.py module, we implement the training workflow. In this section, we describe the training setup and the chosen options for performing the training.
We used the following classes:
- feature extractor (
HOFeatureExtractor), - CRFs model (
HOCRFAD) and - CRFs model representation (
HOCRFADModelRepresentation)
For the training method (i.e. optimization options), we used the following options:
- stochastic gradient ascent with adaptive learning rate (ADADELTA), (Zeiler, 2012) (
method = SGA-ADADELTA), - with
l2regularization(regularization_type = l2)and regularization value equal to 1(regularization_value = 1), - and 15 passes through the training data
num_epochs = 15)
To run the training process, we use run_training(args) function. We pass the optimization options, the function generating the defined feature templates, the target label (i.e. locomotion or gestures) and the train and test file names (see this code snippet). The training process for each task (i.e. task A and B) will perform the following:
- read the training file (
train_discretized_iob.txt) and parse it into sequences - process and dump the parsed sequences on disk in a relevant format to be later used in the learning framework
- build a model based on the processed training sequences
- train the model weights (i.e. estimate the feature weights) using the specified optimization method
- use the trained model to decode the training sequences and write the result to a file
- read the test file (
test_discretized_iob.txt) parse it into sequences - use the trained model to decode the testing sequences and write the result to a file
- return the path to the trained model directory
The return value of the training function (i.e. model_dir -- see code snippet below) is the path to the trained model.
During model training, we track the estimated average loglikelihood by plotting the generated avg_loglikelihood_training file.
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (10,6)
from train_activityrecognizer_workflow import *
# import the module containing training workflow
# we use only 1 sequence for demonstration
# to go through the whole file simply omit passing the num_seqs keyword argument
num_seqs = 1
optimization_options = {"method" : "SGA-ADADELTA",
"regularization_type": "l2",
"regularization_value":1,
"num_epochs":10,
"tolerance":1e-6
}
target_label = 'locomotion'
train_fname = 'train_discretized_iob.txt'
test_fname = 'test_discretized_iob.txt'
# demonstrate training modes of locomotion classifier using only 1 sequence from the both the training and test file
model_dir = run_training(optimization_options,
template_config,
target_label,
train_fname,
test_fname,
num_seqs=num_seqs)
# demonstrate training gestures classifier using only 1 sequence from the both the training and test file
# uncomment the below lines
# target_label = 'gestures'
# model_dir = run_training(optimization_options,
# template_config,
# target_label,
# train_fname,
# test_fname,
# num_seqs=num_seqs)
# using all sequences from the both the training and test file
# optimization_options = {"method" : "SGA-ADADELTA",
# "regularization_type": "l2",
# "regularization_value":1,
# "num_epochs":15,
# "tolerance":1e-6
# }
# model_dir = run_training(optimization_options,
# template_config,
# target_label,
# train_fname,
# test_fname)
from pyseqlab.utilities import ReaderWriter
def plot_avg_loglikelihood(model_dir):
# plot the estimated average loglikelihood during training phase
avg_ll = ReaderWriter.read_data(os.path.join(model_dir, 'avg_loglikelihood_training'))
plt.plot(avg_ll[1:], label="data:{}, method:{}, {}:{}".format(target_label,
optimization_options['method'],
optimization_options['regularization_type'],
optimization_options['regularization_value']))
plt.legend(loc='upper right')
plt.xlabel('number of epochs')
plt.ylabel('estimated average loglikelihood')
plot_avg_loglikelihood(model_dir)
3. Trained model evaluation¶
NB: Before proceeding, we have to unzip the trained_models directory so that we can explore and assess the trained models.
To evaluate the performance of the trained models, we will use the eval_models(args) function in the train_activityrecognizer_workflow.py module. It takes a list of the trained models' path on disk. An example of a trained model directory will have the following structure:
├── avg_loglikelihood_training ├── crf_training_log.txt ├── decoding_seqs │ ├── test_fold_0.txt │ ├── train_fold_0.txt ├── model_parts │ ├── class_desc.txt │ ├── FE_templateX │ ├── FE_templateY │ ├── MR_L │ ├── MR_modelfeatures │ ├── MR_modelfeaturescodebook │ ├── MR_Ycodebook │ ├── weights
Each model has a model_parts folder. The decoded sequences are found under decoding_seqs folder where we have the decoding of the training and test files. We evaluate the performance of the trained models by evaluating the weighted F1-measure using those files as in the code snippet below.
# to evaluate the models' performance on the sequence level, using already trained models
# eval_models takes a list of trained models path on disk
trainedmodels_rootdir = os.path.join(project_dir, 'trained_models')
models_folders = ("2017_3_31-17_1_8_31139", "2017_4_4-18_13_42_852521")
models_dir = [os.path.join(trainedmodels_rootdir, folder) for folder in models_folders]
eval_models(models_dir)
4. Using modes of locomotion classifier (i.e. trained model)¶
In this section, we demonstrate how to revive a previously trained model for identifying modes of locomotion using Opportunity activity recognition dataset.
As a reminder, the trained model (including its components) are found under trained_models folder that has the following structure:
── 2017_3_31-17_1_8_31139 │ ├── crf_training_log.txt │ ├── model_parts │ │ ├── class_desc.txt │ │ ├── MR_L │ │ ├── MR_modelfeatures │ │ ├── MR_modelfeaturescodebook_oldrepr │ │ ├── MR_Ycodebook │ │ ├── FE_templateY │ │ ├── FE_templateX │ │ ├── weights │ │ ├── MR_modelfeaturescodebook │ ├── decoding_seqs │ │ ├── test_fold_0.txt │ │ ├── train_fold_0.txt
4.1 Reviving trained model¶
To use/revive our trained model we use revive_learnedmodel(args) function that takes:
- the path/directory to the trained model
- the attribute extractor class (i.e.
SensorAttributeExtractorCategclass)
The folder 2017_3_31-17_1_8_31139 under trained_models folder represents the path to our trained model.
# we get the trained model parts directory -- check the tree path in the cell above
trained_model_dir = os.path.join(project_dir, 'trained_models')
# loading the trained model
crf_m = revive_learnedmodel(os.path.join(trained_model_dir, '2017_3_31-17_1_8_31139'), SensorAttributeExtractorCateg)
After we have revived our model, we need sequences to decode. We will use the test dataset we already procecessed under the dataset directory. Just as a reminder, the tree path is:
├── dataset
├── locomotion
│ ├── sensor_test_wsize15_stepsize8_yagglast_locomotion.pkl
│ ├── sensor_train_wsize15_stepsize8_yagglast_locomotion.pkl
│ ├── test.txt
│ ├── test_discretized.txt
│ ├── test_discretized_iob.txt
│ ├── train.txt
│ ├── train_discretized.txt
│ ├── train_discretized_iob.txt
To read the test file, we will use DataFileParser class in the utilities module.
from pyseqlab.utilities import DataFileParser
# initialize a data file parser
dparser = DataFileParser()
# initialize a data file parser
dparser = DataFileParser()
# provide the options to parser such as the header info, the separator between words and if the y label is already existing
# main means the header is found in the first line of the file
header = ["s_{}".format(i) for i in range(113)] + ['locomotion']
# y_ref is a boolean indicating if the label to predict is already found in the file
y_ref = True
# spearator between the observations
column_sep = " "
seqs = []
# read all sequences
for seq in dparser.read_file(os.path.join(dataset_dir, 'locomotion', 'test_discretized_iob.txt'), header, y_ref=y_ref, column_sep = column_sep):
seqs.append(seq)
# printing one sequence for display
# print(seqs[0])
print("number of parsed sequences is: ", len(seqs))
4.2 Decoding method¶
Then, we decide the decoding options for our model to use. The main method for decoding is decode_seqs(decoding_method, out_dir, **kwargs) that takes two arguments and multiple keyword arguments.
The obligatory arguments are:
-
decoding_method: string representing the decoding method such as'viterbi' -
output_dir: string, the output directory representing the path where the parsing would take place
For the keyword arguments, the main ones to specify are:
-
seqs: the list of sequences we already parsed/read from the text file we need to label -
file_name: the name of the file where decoded sequences will be written to (it is optional) -
sep: the separator between the columns/observations when writing decoded sequences to the specified file usingfile_namekeyword argument
decoding_method = 'viterbi'
output_dir = os.path.join(project_dir, 'tutorials')
sep = "\t"
# decode sequences
seqs_decoded = crf_m.decode_seqs(decoding_method, output_dir, seqs= seqs, file_name = 'tutorial_seqs_decoding.txt', sep=sep)
The decoded sequences will be found under the tutorials directory (i.e. current directory) under decoding_seqs folder.
|---tutorials | |---decoding_seqs | | |---tutorial_seqs_decoding.txtThe
tutorial_seqs_decoding.txt file will follow the same template/format of the test_discretized_iob.txt file we already parsed earlier, but this time with additional column that contains our model's predictions.
We can check our model performance by using the eval_decoded_file(args) function in the train_activityrecognizer_workflow.py module.
# evaluate model on test data set
new_decseqs_file = os.path.join(tutorials_dir, 'decoding_seqs','tutorial_seqs_decoding.txt')
eval_decoded_file(new_decseqs_file, other_symbol='0')
Our model achieves overall weighted F1-measure equal to 89%
Although, the trained models clearly overfit the training data (i.e. the F1-scores are beyond the 97%), we obtained a very good performance using the test data. Clearly, the effort should be on experimenting with various feature templates and regularization values, various model structures (i.e. smaller models) even different representation of the training and testing sequences (i.e. fixed length sequences). We leave this as an exercise for the readers ... :)
5. Literature and references¶
Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Gerhard Tröster, Paul Lukowicz, Gerald Pirkl, David Bannach, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha, Hamidreza Bayati, and José del R. Millàn. "Collecting complex activity data sets in highly rich networked sensor environments" In Seventh International Conference on Networked Sensing Systems (INSS’10), Kassel, Germany, 2010.
Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundaratejaswi Digumarti, Gerhard Tröster, José del R. Millán, Daniel Roggen. "The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition", Pattern Recognition Letters, 2013
Raymond, C. (2013). Robust tree-structured Named Entities Recognition from speech. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 8475–8479). IEEE. http://doi.org/10.1109/ICASSP.2013.6639319
Fayyad, U. M., & Irani, K. B. (1993). Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning. In Proceedings of the International Joint Conference on Uncertainty in AI (pp. 1022–1027). Retrieved from http://trs-new.jpl.nasa.gov/dspace/handle/2014/35171