IDENTIFYING ACTIVITIES USING SENSOR DATA

1 . A method for identifying activities, comprising:

obtaining a training set, wherein the training set comprises a plurality of training samples, a training sample of the plurality of training samples comprising: a sequence of sensor data obtained from one or more sensors disposed on a user spanning a first time duration, and a label of an activity the user was engaged in during collection of the sequence of sensor data;

training a two-stage neural network to generate, for each training sample in the training set, an output indicating a corresponding label of the activity, wherein the two-stage neural network comprises:

a plurality of low-level encoders configured to receive a subset of the sequence of sensor data spanning a second time duration as input and generate a plurality of low-level outputs, the plurality of low-level encoders configured to each receive a different portion of the subset of the sequence of sensor data, the plurality of low-level outputs predicting a low-level motion pattern associated with a respective different portion of the subset of the sequence of sensor data, and the second time duration shorter than the first time duration, and

a high-level encoder configured to receive the plurality of low-level outputs and generate the output indicating the corresponding label of the activity, wherein the respective predicted low-level motion patterns are subsets of high-level motion patterns associated with the activity; and

providing one or more parameters associated with a trained two-stage neural network to a user device, such that the user device uses the one or more parameters to identify activities based on sensor data.

1 . The method of claim 1 , wherein the second time duration is less than about two seconds.

2 . The method of claim 1 , wherein the first time duration is more than about 30 seconds.

3 . The method of claim 1 , wherein the low-level encoders of the plurality of low-level encoders include one or more of a fully connected network, a recurrent neural network, a long short-term (LSTM) network, a gated recurrent unit (GRU) network, a one dimensional convolutional neural network (1-D CNN), or a temporal convolutional network (TCN).

4 . The method of claim 1 , wherein the high-level encoder is a fully connected network, a recurrent neural network, a long short-term (LSTM) network, a gated recurrent unit (GRU) network, a one dimensional convolutional neural network (1-D CNN), or a temporal convolutional network (TCN).

5 . The method of claim 1 , wherein training the two-stage neural network comprises:

partitioning the sequence of sensor data into the different portions of the subset of the sequence of sensor data based on a plurality of time windows; and

providing each of the different portions to the plurality of low-level encoders, wherein the plurality of different portions are non-overlapping.

6 . The method of claim 5 , wherein at least two time windows of the plurality of time windows have different durations.

7 . The method of claim 1 , wherein training the two-stage neural network comprises:

determining, for a training sample in the training set, an error associated with a predicted activity generated by the high-level encoder relative to the corresponding label of the activity; and

updating weights for the low-level encoder and the high-level encoder based on the error.

8 . A method for identifying activities, comprising:

obtaining a sequence of sensor data obtained from one or more sensors disposed on a user, the sequence of sensor data spanning a first time duration;

partitioning the sequence of sensor data into a plurality of subsets of sensor data, each subset of sensor data being different and spanning a time duration less than the first time duration;

providing each subset of the plurality of subsets of sensor data to a low-level encoder of a plurality of low-level encoders, wherein the low-level encoders generate, for the plurality of subsets of sensor data, a plurality of low-level outputs corresponding to the plurality of subsets of sensor data, and the plurality of low-level outputs predict a low-level motion pattern associated with a respective different subset of the sequence of sensor data; and

providing the plurality of low-level outputs to a high-level encoder to generate a prediction of an activity the user was engaged in during collection of the sequence of sensor data, wherein the respective predicted low-level motion patterns are subsets of high-level motion patterns associated with the activity, and wherein the low-level encoder and the high-level encoder were both trained using a training set comprising sequences of sensor data spanning time durations greater than the time duration associated with each subset of sensor data.

9 . The method of claim 8 , further comprising identifying at least one action to be performed by a user device associated with the one or more sensors based on the prediction of the activity the user was engaged in during the collection of the sequence of sensor data.

10 . The method of claim 9 , wherein the at least one action comprises:

causing information relevant to the activity the user was engaged in to be presented, causing a playlist of media content items to begin being presented, causing a pre-defined scripted set of activities to be executed.

11 . A system for identifying activities, the system comprising:

a memory; and

one or more processors communicatively coupled with the memory, the one or more processors configured to:

obtain a training set, wherein the training set comprises a plurality of training samples, a training sample of the plurality of training samples comprising: a sequence of sensor data obtained from one or more sensors disposed on a user spanning a first time duration, and a label of an activity the user was engaged in during collection of the sequence of sensor data;

train a two-stage neural network to generate, for each training sample in the training set, an output indicating a corresponding label of the activity, wherein the two-stage neural network comprises:

a plurality of low-level encoders configured to receive a subset of the sequence of sensor data spanning a second time duration as input and generate a plurality of low-level outputs, the plurality of low-level encoders configured to each receive a different portion of the subset of the sequence of sensor data, the plurality of low-level outputs predicting a low-level motion pattern associated with a respective different portion of the subset of the sequence of sensor data, and the second time duration shorter than the first time duration, and

a high-level encoder configured to receive the plurality of low-level outputs and generate the output indicating the corresponding label of the activity, wherein the respective predicted low-level motion patterns are subsets of high-level motion patterns associated with the activity; and

provide one or more parameters associated with a trained two-stage neural network to a user device, such that the user device uses the one or more parameters to identify activities based on sensor data.

12 . The system of claim 11 , wherein the sequence of sensor data comprises at least one of: accelerometer data, gyroscope data, pressure sensor data, magnetometer data, or ambient light senor data.

13 . The system of claim 11 , wherein to train the two-stage network, the one or more processors are further configured to:

determine, for a training sample in the training set, an error associated with a predicted activity generated by the high-level encoder relative to the corresponding label of the activity; and

update weights for the low-level encoder and the high-level encoder based on the error.

14 . A system for identifying activities, the system comprising:

a memory; and

one or more processors communicatively coupled to the memory, the one or more processors configured to:

obtain a sequence of sensor data obtained from one or more sensors disposed on a user, the sequence of sensor data spanning a first time duration;

partition the sequence of sensor data into a plurality of subsets of sensor data, each subset of sensor data being different and spanning a time duration less than the first time duration;

provide each subset of the plurality of subsets of sensor data to a low-level encoder of a plurality of low-level encoders, wherein the low-level encoders generate, for the plurality of subsets of sensor data, a plurality of low-level outputs corresponding to the plurality of subsets of sensor data, and the plurality of low-level outputs predict a low-level motion pattern associated with a respective different subset of the sequence of sensor data; and

provide the plurality of low-level outputs to a high-level encoder to generate a prediction of an activity the user was engaged in during collection of the sequence of sensor data, wherein the respective predicted low-level motion patterns are subsets of high-level motion patterns associated with the activity, and wherein the low-level encoder and the high-level encoder were both trained using a training set comprising sequences of sensor data spanning time durations greater than the time duration associated with each subset of sensor data.

15 . The system of claim 14 , wherein the one or more processors are further configured to identify at least one action to be performed by a user device associated with the one or more sensors based on the prediction of the activity the user was engaged in during the collection of the sequence of sensor data.

16 . The system of claim 15 , wherein the at least one action comprises:

causing information relevant to the activity the user was engaged in to be presented, causing a playlist of media content items to begin being presented, causing a pre-defined scripted set of activities to be executed.

17 . The system of claim 14 , wherein the one or more sensors are disposed at different locations on a body of the user, and wherein the different locations comprise: a head of the user, a wrist of the user, a finger of the user, a torso of the user, a foot of the user, and/or a leg of the user.

18 . The system of claim 14 , wherein the one or more sensors are embedded into a wearable device.

19 . The system of claim 19 , wherein the low-level encoder is configured to execute on the wearable device, and the high-level encoder is configured to execute on a different device than the wearable device.