Federated on-sensor learning for local device adaptation and global improvement of machine learning model
An artificial reality system performs local user adaptation of machine learning models and global improvement of the machine learning models while ensuring data security and privacy. An artificial reality headset uses smart sensors and a system on chip (SoC). The smart sensor extracts features from sensor data, encrypts them and transmits the encrypted features to the SoC of the headset. The SoC provides the encrypted features to a server, for example, a server in a cloud platform. The server updates a base model with collected features and sends the updated base model parameters to headsets. The smart sensors of a headset receiving the updated model parameters, balance the model parameters with locally stored feature data, for example, user calibration features to perform user customization of the machine learning model.
1 . A method comprising:
receiving, by a smart sensor of an artificial reality headset, parameters of a machine learning model;
updating, by the smart sensor, the machine learning model based on feature data extracted from the sensor data; and
repeatedly performing:
sending, by the smart sensor, encrypted data associated with the machine learning model to a system on chip (SoC) of the headset;
receiving, by the SoC, encrypted data associated with the machine learning model from one or more smart sensors;
sending, by the SoC, encrypted data associated with the machine learning model collected from the one or more smart sensors to a server;
receiving, by the SoC, parameters of a base machine learning model from the server, and
sending, by the SoC, parameters of the base machine learning model to the smart sensor; and
locally updating, by the smart sensor, the received base machine learning model based on the stored feature data.
2 . The method of claim 1 , wherein encrypted data associated with the machine learning model represents encrypted feature data.
3 . The method of claim 1 , wherein encrypted data associated with the machine learning model represents encrypted parameters of the updated machine learning model.
4 . The method of claim 3 , wherein the server generates the base model by aggregating parameters received from a plurality of artificial reality headsets.
5 . The method of claim 1 , wherein locally updating comprises calibrating the machine learning model for a specific user.
6 . The method of claim 1 , wherein smart sensors and SoC are part of an artificial reality headset.
7 . The method of claim 1 , wherein the server:
receives encrypted features from a plurality of headsets,
decrypts the encrypted features, and
updates a base machine learning model based on the decrypted features.
8 . The method of claim 1 , wherein the parameters of the machine learning model are received from the server as part of an initial deployment.
9 . The method of claim 1 , wherein the SoC stores parameters received form one or more smart sensors for backup.
10 . The method of claim 1 , wherein one or more smart sensors include eye tracking sensors and hand tracking sensors.
11 . A non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:
receiving, by a smart sensor of an artificial reality headset, parameters of a machine learning model;
updating, by the smart sensor, the machine learning model based on feature data extracted from the sensor data; and
repeatedly performing:
sending, by the smart sensor, encrypted data associated with the machine learning model to a system on chip (SoC) of the headset;
receiving, by the SoC, encrypted data associated with the machine learning model from one or more smart sensors;
sending, by the SoC, encrypted data associated with the machine learning model collected from the one or more smart sensors to a server;
receiving, by the SoC, parameters of a base machine learning model from the server, and
sending, by the SoC, parameters of the base machine learning model to the smart sensor; and
locally updating, by the smart sensor, the received base machine learning model based on the stored feature data.
12 . The non-transitory computer readable storage medium of claim 11 , wherein encrypted data associated with the machine learning model represents encrypted feature data.
13 . The non-transitory computer readable storage medium of claim 11 , wherein encrypted data associated with the machine learning model represents encrypted parameters of the updated machine learning model.
14 . The non-transitory computer readable storage medium of claim 13 , wherein the server generates the base model by aggregating parameters received from a plurality of artificial reality headsets.
15 . The non-transitory computer readable storage medium of claim 11 , wherein locally updating comprises calibrating the machine learning model for a specific user.
16 . The non-transitory computer readable storage medium of claim 11 , wherein smart sensors and SoC are part of an artificial reality headset.
17 . The non-transitory computer readable storage medium of claim 11 , wherein the server:
receives encrypted features from a plurality of headsets,
decrypts the encrypted features, and
updates a base machine learning model based on the decrypted features.
18 . The non-transitory computer readable storage medium of claim 11 , wherein the parameters of the machine learning model are received from the server as part of an initial deployment.
19 . The non-transitory computer readable storage medium of claim 11 , wherein the SoC stores parameters received form one or more smart sensors for backup.
20 . The non-transitory computer readable storage medium of claim 11 , wherein one or more smart sensors include eye tracking sensors and hand tracking sensors.