Time-of-flight wake on attention
According to an embodiment, a method includes collecting, by a time-of-flight (ToF) sensor, real time raw data of a scene comprising a user in its field of view; generating, by a first neural network, an estimate of a posture of the user based on a distance calculated to the user; generating, by a second neural network and in response to determining that the estimate of the posture of the user meets a first criterion, an estimate of a head orientation of the user based on the distance, a standard deviation of the distance, the number of photons returning per unit of time, a standard deviation of the signal value, and a reflection value; and transitioning from low-power to normal mode in response to determining that the user is actively engaged with the device based on the estimate of the head orientation of the user.
1 . A device, comprising:
a time-of-flight (ToF) sensor configured to collect real time raw data of a scene comprising a user in a field of view of the ToF sensor; and
a processor comprising a first neural network and a second neural network,
wherein the first neural network is configured to generate an estimate of a posture of the user based on a distance calculated for each zone of the ToF sensor to the user,
wherein the second neural network is configured to, in response to determining by the processor that the estimate of the posture of the user meets a first criterion, generate an estimate of a head orientation of the user based on the distance, a standard deviation of the distance, a signal value corresponding to a number of photons returning towards the ToF sensor per unit of time, the a standard deviation of the signal value, and a reflection value calculated for each zone of the ToF sensor, and
wherein the processor is configured to transition the device from a low-power operating mode to a normal operating mode in response to determining that the user is actively engaged with the device based on the estimate of the head orientation of the user.
2 . The device of claim 1 , wherein the processor is further configured to determine a duration that the user is actively engaged with the device, wherein the processor is configured to transition the device from a low-power operating mode to a normal operating mode in response to determining that the duration the user is actively engaged with the device based on the estimate of the head orientation of the user exceeds a threshold.
3 . The device of claim 1 , wherein the estimate of the head orientation of the user is a direction describing an orientation of a face of the user, among the North, North-East, North-West, East, West, and South directions, the North direction indicating that the user is facing the ToF sensor, and the South direction indicating that the user has its back facing the ToF sensor.
4 . The device of claim 3 , wherein an output layer of the second neural network includes six nodes, each node corresponding to one of the North, North-East, North-West, East, West, and South directions, wherein the node corresponding to the North direction equals “1” when the user is facing the ToF sensor and the other nodes are equal to “0”.
5 . The device of claim 1 , wherein the first criterion comprises the user sitting within a threshold distance from the device and the head of the user is within the field of view of the ToF sensor.
6 . The device of claim 1 , wherein the estimate of the head orientation of the user is a Boolean value having a value of “0,” indicating the user is not actively engaged with the device or a value of “1,” indicating that the user is actively engaged with the device.
7 . The device of claim 1 , wherein the estimate of the head orientation of the user is a value within a range of values, wherein the value at a first extreme of the range of values indicates the user's head is fully rotated away from the device and a value at the other extreme of the range of values indicates that a face of the user is directly positioned in front of the device and centered within the field of view of the ToF sensor.
8 . A method, comprising:
collecting, by a time-of-flight (ToF) sensor of a device, real time raw data of a scene comprising a user in a field of view of the ToF sensor;
generating, by a first neural network, an estimate of a posture of the user based on a distance calculated for each zone of the ToF sensor to the user;
generating, by a second neural network and in response to determining by a processor that the estimate of the posture of the user meets a first criterion, an estimate of a head orientation of the user based on the distance, a standard deviation of the distance, a signal value corresponding to a number of photons returning towards the ToF sensor per unit of time, a standard deviation of the signal value, and a reflection value calculated for each zone of the ToF sensor to the user; and
transitioning, by the processor, the device from a low-power operating mode to a normal operating mode in response to determining that the user is actively engaged with the device based on the estimate of the head orientation of the user.
9 . The method of claim 8 , further comprising determining, by the processor, a duration that the user is actively engaged with the device, wherein the determining that the user is actively engaged with the device comprises determining that the duration the user is actively engaged with the device based on the estimate of the head orientation of the user exceeds a threshold.
10 . The method of claim 8 , wherein the estimate of the head orientation of the user is a direction describing an orientation of a face of the user, among the North, North-East, North-West, East, West, and South directions, the North direction indicating that the user is facing the ToF sensor, and the South direction indicating that the user has its back facing the ToF sensor.
11 . The method of claim 10 , wherein an output layer of the second neural network includes six nodes, each node corresponding to one of the North, North-East, North-West, East, West, and South directions, wherein the node corresponding to the North direction equals “1” when the user is facing the ToF sensor and the other nodes are equal to “0”.
12 . The method of claim 8 , wherein the first criterion comprises the user sitting within a threshold distance from the device and the head of the user is within the field of view of the ToF sensor.
13 . The method of claim 8 , wherein the estimate of the head orientation of the user is a Boolean value having a value of “0,” indicating the user is not actively engaged with the device or a value of “1,” indicating that the user is actively engaged with the device.
14 . The method of claim 8 , wherein the estimate of the head orientation of the user is a value within a range of values, wherein the value at a first extreme of the range of values indicates the users head is fully rotated away from the device and a value at the other extreme of the range of values indicates that a face of the user is directly positioned in front of the device and centered within the field of view of the ToF sensor.
15 . A method, comprising:
collecting, by a time-of-flight (ToF) sensor of a device, raw data of scenes in a field of view of the ToF sensor, wherein the scenes correspond to different rotations, orientations, positionings of a user in the field of view of the ToF sensor;
training a neural network using the collected raw data;
collecting, by the ToF, real time raw data of a scene comprising a user in the field of view of the ToF sensor;
generating, by the neural network, an estimate of a posture of the user based on a distance calculated for each zone of the ToF sensor;
generating, by the neural network and in response to determining by a processor that the estimate of the posture of the user meets a first criterion, an estimate of a head orientation of the user based on the distance, a standard deviation of the distance, a signal value corresponding to a number of photons returning towards the ToF sensor per unit of time, and a standard deviation of the signal value for each zone of the ToF sensor to the user; and
activating, by the processor, an authentication procedure in response to determining that the user is actively engaged with the device based on the estimate of the head orientation of the user.
16 . The method of claim 15 , wherein the generating the estimate of the head orientation further comprises generating the estimate of the head orientation based on a reflection value calculated for each zone of the ToF sensor.
17 . The method of claim 15 , wherein the estimate of the head orientation of the user is a direction describing an orientation of a face of the user, among the North, North-East, North-West, East, West, and South directions, the North direction indicating that the user is facing the ToF sensor, and the South direction indicating that the user has its back facing the ToF sensor.
18 . The method of claim 15 , wherein an output layer of the neural network includes six nodes, each node corresponding to one of the North, North-East, North-West, East, West, and South directions, wherein the node corresponding to the North direction equals “1” when the user is facing the ToF sensor and the other nodes are equal to “0”.
19 . The method of claim 15 ,
wherein the estimate of the head orientation of the user is a Boolean value having a value of “0,” indicating the user is not actively engaged with the device or a value of “1,” indicating that the user is actively engaged with the device, or
wherein the estimate of the head orientation of the user is a value within a range of values, wherein the value at a first extreme of the range of values indicates the users head is fully rotated away from the device and a value at the other extreme of the range of values indicates that a face of the user is directly positioned in front of the device and centered within the field of view of the ToF sensor.
20 . The method of claim 15 , wherein the first criterion comprises the user sitting within a threshold distance from the device and the head of the user is within the field of view of the ToF sensor.