Methods and apparatus for actions, activities and tasks classifications based on machine learning techniques
Systems and methods of the present disclosure enable automated recognition of user performed activities and tasks using sensor data by receiving raw sensor data while a user performs a series of activities wearing at least one sensor for a predetermined interval of time. The raw sensor data is converted into a set of feature values. An action recognition machine learning model is used to generate action labels indicative of actions performed by the user during the predetermined interval of time based on trained action model parameters and the set of feature values. A task recognition machine learning model is used to generate task labels indicative of tasks performed by the user during the predetermined interval of time based on trained task model parameters, the set of action labels and the set of feature values, and a message is displayed with an indication of the task labels to a user.
1. An apparatus, comprising:
a processor; and
a non-transitory memory storing instructions which, when executed by the processor, causes the processor to:
capture raw sensor data while a user performs a series of activities wearing at least one sensor;
wherein the raw sensor data is for a predetermined interval of time;
wherein the predetermined interval of time comprises a series of timestamps;
wherein the raw sensor data comprises at least one movement measurement at each timestamp in the series of timestamps in the predetermined interval of time;
convert the raw sensor data into a set of feature values;
utilize an action recognition machine learning model to generate a set of action labels for the predetermined interval of time based on trained action model parameters and the set of feature values;
wherein the set of action labels are indicative of at least one action category of at least one action performed by the user during the predetermined interval of time;
utilize an activity recognition machine learning model to generate a set of activity labels for the predetermined interval of time based at least in part on trained activity model parameters and the set of action labels;
wherein the set of activity labels are indicative of at least one activity performed by the user during the predetermined interval of time;
wherein the at least one activity represents at least one action sequence comprising the at least one action performed by the user during the predetermined interval of time;
utilize a task recognition machine learning model to generate a set of task labels for the predetermined interval of time based at least in part on trained task model parameters, the set of action labels and the set of activity labels;
wherein the set of task labels are indicative of at least one task category of at least one task performed by the user during the predetermined interval of time;
wherein the at least one task is associated with at least one of:
the at least one action sequence comprising the at least one action performed by the user during the predetermined interval of time, and
at least one activity sequence comprising the at least one activity performed by the user during the predetermined interval of time; and
transmit, to a computing device associated with the user, a message comprising an indication representative of the set of task labels indicative of the at least one task.
2. The apparatus as recited in claim 1 , wherein the activity recognition machine learning model comprises a multi-classification machine learning model.
3. The apparatus as recited in claim 1 , wherein the action recognition machine learning model comprises a multi-classification machine learning model.
4. The apparatus as recited in claim 1 , wherein the task recognition machine learning model comprises a multi-classification machine learning model.
5. The apparatus as recited in claim 1 , wherein the instructions, when executed by the processor, further cause the processor to determine at least one time-domain feature based at least in part on a timestamp associated with each raw sensor data record of the raw sensor data.
6. The apparatus as recited in claim 1 , wherein the instructions, when executed by the processor, further cause the processor to determine at least one frequency-domain feature based at least in part on a frequency associated with each raw sensor data record of the raw sensor data.
7. The apparatus as recited in claim 1 , wherein the instructions, when executed by the processor, further cause the processor to:
generate a plurality of windows of data;
wherein each window of data of the plurality of windows of data comprises a fixed interval of time that is less than the predetermined interval of time;
wherein each window of data of the plurality of windows of data overlaps with a subsequent window of data of the plurality of windows of data to form overlapping windows of data; and
convert the raw sensor data into a subset of feature values of the set of feature values for each window of data of the plurality of windows of data.
8. The apparatus as recited in claim 1 , wherein the raw sensor data comprises sensor measurements captured at a 12.5 Hertz (Hz) frequency.
9. The apparatus as recited in claim 1 , wherein each sensor measure of the sensor measurements comprise at least one of:
i) a timestamp,
ii) at least one yaw measurement,
iii) at least one pitch measurement,
iv) at least one roll measurement,
v) at least one x-axis acceleration (acc_x) measurement,
vi) at least one y-axis acceleration (acc_y) measurement,
vii) at least one z-axis acceleration (acc_z) measurement,
viii) at least one x-axis gyroscopic (gyro_x) measurement,
ix) at least one y-axis gyroscopic (gyro_y) measurement, and
x) at least one z-axis gyroscopic (gyro_z) measurement.
10. A method, comprising:
receiving, by a processor, raw sensor data while a user performs a series of activities wearing at least one sensor;
wherein the raw sensor data is for a predetermined interval of time;
wherein the predetermined interval of time comprises a series of timestamps;
wherein the raw sensor data comprises at least one movement measurement at each timestamp in the series of timestamps in the predetermined interval of time;
converting, by a processor, the raw sensor data into a set of feature values;
utilizing, by a processor, an action recognition machine learning model to generate a set of action labels for the predetermined interval of time based on trained action model parameters and the set of feature values;
wherein the set of action labels are indicative of at least one action category of at least one action performed by the user during the predetermined interval of time;
utilize an activity recognition machine learning model to generate a set of activity labels for the predetermined interval of time based at least in part on trained activity model parameters and the set of action labels;
wherein the set of activity labels are indicative of at least one activity performed by the user during the predetermined interval of time;
wherein the at least one activity represents at least one action sequence comprising the at least one action performed by the user during the predetermined interval of time;
utilizing, by a processor, a task recognition machine learning model to generate a set of task labels for the predetermined interval of time based at least in part on trained task model parameters, the set of action labels and the set of activity labels;
wherein the set of task labels are indicative of at least one task category of at least one task performed by the user during the predetermined interval of time;
wherein the at least one task is associated with at least one of:
the at least one action sequence comprising the at least one action performed by the user during the predetermined interval of time, and
at least one activity sequence comprising the at least one activity performed by the user during the predetermined interval of time; and
transmitting, by a processor, to a computing device associated with the user, a message comprising an indication representative of the set of task labels indicative of the at least one task.
11. The method as recited in claim 10 , wherein the activity recognition machine learning model comprises a multi-classification machine learning model.
12. The method as recited in claim 10 , wherein the action recognition machine learning model comprises a multi-classification machine learning model.
13. The method as recited in claim 10 , wherein the task recognition machine learning model comprises a multi-classification machine learning model.
14. The method as recited in claim 10 , further comprising determining, by a processor, at least one time-domain feature based at least in part on a timestamp associated with each raw sensor data record of the raw sensor data.
15. The method as recited in claim 10 , further comprising determining, by a processor, at least one frequency-domain feature based at least in part on a frequency associated with each raw sensor data record of the raw sensor data.
16. The method as recited in claim 10 , further comprising:
generating, by a processor, a plurality of windows of data;
wherein each window of data of the plurality of windows of data comprises a fixed interval of time that is less than the predetermined interval of time;
wherein each window of data of the plurality of windows of data overlaps with a subsequent window of data of the plurality of windows of data to form overlapping windows of data; and
converting, by a processor, the raw sensor data into a subset of feature values of the set of feature values for each window of data of the plurality of windows of data.
17. The method as recited in claim 10 , wherein the raw sensor data comprises sensor measurements captured at a 12.5 Hertz (Hz) frequency.
18. The method as recited in claim 17 , wherein each sensor measurement of the sensor measurements comprise at least one of:
i) a timestamp,
ii) at least one yaw measurement,
iii) at least one pitch measurement,
iv) at least one roll measurement,
v) at least one x-axis acceleration (acc_x) measurement,
vi) at least one y-axis acceleration (acc_y) measurement,
vii) at least one z-axis acceleration (acc_z) measurement,
viii) at least one x-axis gyroscopic (gyro_x) measurement,
ix) at least one y-axis gyroscopic (gyro_y) measurement, and
x) at least one z-axis gyroscopic (gyro_z) measurement.