Runtime assessment of sensors
This relates to the use of sensor evaluation in a multi-sensor environment. In a first aspect, this specification describes apparatus comprising: at least one processor; and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: receive sensor data from a plurality of sensors collected during a first time period; process the received sensor data through a plurality of layers of a neural network to generate an output indicative of the sensing quality of each of the plurality of sensors for a task; and cause a subset of the plurality of sensors to collect data during a second time period based on the output indicative of the suitability of each of the plurality of sensors for the task.
1 . An apparatus comprising at least one processor; and
at least one memory including computer program code;
the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform:
generate ground truth quality data for a plurality of sensors, the ground truth quality data comprising sensor data collected by the plurality of the sensors and a corresponding quality value for an inference task, wherein the corresponding quality value indicates an accuracy of the inference task using the sensor data;
process the sensor data from the ground truth quality data through a plurality of layers of a neural network to generate an output indicative of sensing quality of the plurality of the sensors for the inference task;
update weights of the neural network in dependence on a comparison of the output indicative of the sensing quality of the plurality of sensors to the quality values of the ground truth quality data;
select one or more of the plurality of the sensors based on the output indicative of the sensing quality of the one or more of the plurality of the sensors for the inference task;
cause the one or more of the plurality of the sensors to collect data during a second time period for the inference task; and
apply a sensing model of the selected one or more sensors to the collected data to generate a probability distribution over a plurality of classes for the inference task.
2 . The apparatus of claim 1 , wherein the generating of the ground truth quality data for the plurality of the sensors comprises, for one or more sensors in the plurality of the sensors:
apply a sensing model to sensor data collected from a sensor in the plurality of the sensors to generate a predicted class for said sensor data; and
generate the ground truth quality data in dependence on the comparison of the predicted class for said sensor data to a known class for said sensor data.
3 . The apparatus of claim 1 , wherein operations of the processing of the sensor data from the ground truth quality data and the updating of the weights of the neural network are iterated until a threshold condition is met.
4 . The apparatus of claim 1 , wherein the comparison of the output indicative of the sensing quality of the plurality of the sensors for the inference task to the quality values of the ground truth quality data is performed using a loss function.
5 . The apparatus of claim 4 , wherein the weights are determined by applying an optimization procedure to the loss function.
6 . The apparatus of claim 1 , wherein the neural network comprises:
a plurality of input sub-networks, each input sub-network configured to receive as input sensor data from one of the plurality of the sensors and to extract one or more features from said input sensor data;
a plurality of fully connected layers configured to process features extracted by the plurality of the input sub-networks and generate the output indicative of the suitability of each of the plurality of the sensors for the inference task.
7 . The apparatus of claim 6 , wherein two or more of the input sub-networks have identical weights.
8 . The apparatus of claim 1 , wherein the output indicative of the suitability of the plurality of the sensors for the inference task is a set of binary values, each binary value associated with one of the sensors in the plurality of the sensors and indicative of the suitability of said one of the sensors for performing the inference task.
9 . The apparatus of claim 1 , wherein the apparatus is a user device, a smartphone, a smartwatch, a smart earbud, a tablet device or a server.
10 . The apparatus of claim 1 , wherein the plurality of the sensors are implemented in one or more of the user device, the smartphone, the smartwatch, the smart earbud, the tablet device or the server.
11 . The apparatus of claim 1 , wherein the inference task comprises one or more of human activity recognition, hot-word recognition, health monitoring, environmental monitoring, physiological monitoring, and/or exercise monitoring.
12 . A method comprising:
generating ground truth quality data for a plurality of sensors, the ground truth quality data comprising sensor data collected by the plurality of the sensors and a corresponding quality value for an inference task, wherein the corresponding quality value indicates an accuracy of the inference task using the sensor data;
processing the sensor data from the ground truth quality data through a plurality of layers of a neural network to generate an output indicative of sensing quality of the plurality of the sensors for the inference task;
updating weights of the neural network in dependence on a comparison of the output indicative of the sensing quality of the plurality of sensors to the quality values of the ground truth quality data;
selecting one or more of the plurality of the sensors based on the output indicative of the sensing quality of the one or more of the plurality of the sensors for the inference task;
causing the one or more of the plurality of the sensors to collect data during a second time period for the inference task; and
applying a sensing model of the selected one or more sensors to the collected data to generate a probability distribution over a plurality of classes for the inference task.
13 . The method of claim 12 , wherein the generating of the ground truth quality data for the plurality of the sensors comprises, for one or more sensors in the plurality of the sensors:
applying a sensing model to sensor data collected from a sensor in the plurality of the sensors to generate a predicted class for said sensor data; and
generating the ground truth quality data in dependence on the comparison of the predicted class for said sensor data to a known class for said sensor data.
14 . The method of claim 12 , wherein operations of the processing of the sensor data from the ground truth quality data and the updating of the weights of the neural network are iterated until a threshold condition is met.
15 . The method of claim 12 , wherein the comparison of the output indicative of the sensing quality of the plurality of the sensors for the inference task to the quality values of the ground truth quality data is performed using a loss function.
16 . The method of claim 15 , wherein the weights are determined by applying an optimization procedure to the loss function.
17 . The method of claim 12 , wherein the neural network comprises:
a plurality of input sub-networks, each input sub-network configured to receive as input sensor data from one of the plurality of the sensors and to extract one or more features from said input sensor data;
a plurality of fully connected layers configured to process features extracted by the plurality of the input sub-networks and generate the output indicative of the suitability of each of the plurality of the sensors for the inference task.
18 . A non-transitory computer readable medium comprising program instructions for causing an apparatus to perform:
generating ground truth quality data for a plurality of sensors, the ground truth quality data comprising sensor data collected by the plurality of the sensors and a corresponding quality value for an inference task, wherein the corresponding quality value indicates an accuracy of the inference task using the sensor data;
processing the sensor data from the ground truth quality data through a plurality of layers of a neural network to generate an output indicative of sensing quality of the plurality of the sensors for the inference task;
updating weights of the neural network in dependence on a comparison of the output indicative of the sensing quality of the plurality of sensors to the quality values of the ground truth quality data;
selecting one or more of the plurality of the sensors based on the output indicative of the sensing quality of the one or more of the plurality of the sensors for the inference task;
causing the one or more of the plurality of the sensors to collect data during a second time period for the inference task; and
applying a sensing model of the selected one or more sensors to the collected data to generate a probability distribution over a plurality of classes for the inference task.
19 . The apparatus of claim 1 , wherein a sensor of the plurality of sensors is associated with a respective sensor model.