On-device monitoring and analysis of on-device machine learning model drift
A method includes obtaining a pre-trained machine learning model and a training embedding snapshot from a remote system, and obtaining one or more input data samples captured by a user device. The method includes, for each particular input data sample of the one or more input data samples, processing, using an on-device machine learning model corresponding to the pre-trained machine learning model, the particular input data sample to generate a corresponding on-device embedding and one or more corresponding predicted outputs, and generating, using the training embedding snapshot and the corresponding on-device embedding, corresponding performance data. The method includes aggregating the corresponding performance data for the one or more input data samples to determine one or more performance metrics for the on-device machine learning model, and transmitting the one or more performance metrics to the remote system.
1 . A method comprising:
obtaining, by data processing hardware of a user device, a pre-trained machine learning model and a training embedding snapshot from a remote system;
obtaining, by the data processing hardware, one or more input data samples captured by the user device;
for each particular input data sample of the one or more input data samples:
processing, by the data processing hardware and using an on-device machine learning model corresponding to the pre-trained machine learning model, the particular input data sample to generate a corresponding on-device embedding and one or more corresponding predicted outputs; and
generating, by the data processing hardware and based on the training embedding snapshot and the corresponding on-device embedding, corresponding performance data;
aggregating, by the data processing hardware, the corresponding performance data for the one or more input data samples to determine one or more performance metrics for the on-device machine learning model; and
causing the remote system to analyze a performance of the on-device machine learning model by, at least, transmitting, by the data processing hardware, the one or more performance metrics to the remote system without transmitting the one or more input data samples to the remote system.
2 . The method of claim 1 , wherein the one or more performance metrics are generated without exposing content of the corresponding on-device embeddings or the corresponding predicted outputs to the remote system.
3 . The method of claim 1 , wherein the one or more performance metrics represent a drift in one or more characteristics of the one or more input data samples over time.
4 . The method of claim 1 , wherein the one or more performance metrics represent a drift in the on-device machine learning model over time.
5 . The method of claim 1 , further comprising: receiving, by the data processing hardware, a trigger from the remote system, wherein generating the corresponding performance data and transmitting the one or more performance metrics to the remote system are performed in response to receiving the trigger.
6 . The method of claim 5 , wherein the trigger comprises a cloud messaging push notification.
7 . The method of claim 5 , wherein the trigger comprises logic for generating the corresponding performance data.
8 . The method of claim 5 , wherein the trigger comprises at least one of an indication of a time period over which the corresponding performance data is to be generated, an indication of how often the corresponding performance data is to be generated, or an indication of how often the corresponding performance data is to be aggregated and reported.
9 . The method of claim 1 , wherein generating, based on the training embedding snapshot and the corresponding on-device embedding, the corresponding performance data comprises determining one or more differences between the training embedding snapshot and the corresponding on-device embedding.
10 . The method of claim 9 , wherein the one or more differences comprise a cosine similarity or a Euclidean distance.
11 . A system comprising:
data processing hardware of a user device; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to:
obtain a pre-trained machine learning model and a training embedding snapshot from a remote system;
obtain one or more input data samples captured by the user device;
for each particular input data sample of the one or more input data samples:
process, using an on-device machine learning model corresponding to the pre-trained machine learning model, the particular input data sample to generate a corresponding on-device embedding and one or more corresponding predicted outputs; and
generate, using the training embedding snapshot and the corresponding on-device embedding, corresponding performance data;
aggregate the corresponding performance data for the one or more input data samples to determine one or more performance metrics for the on-device machine learning model; and
cause the remote system to analyze a performance of the on-device machine learning model by, at least, transmitting the one or more performance metrics to the remote system without transmitting the one or more input data samples to the remote system.
12 . The system of claim 11 , wherein the one or more performance metrics are generated without exposing content of the corresponding on-device embeddings or the corresponding predicted outputs to the remote system.
13 . The system of claim 11 , wherein the one or more performance metrics represent a drift in one or more characteristics of the one or more input data samples over time.
14 . The system of claim 11 , wherein the one or more performance metrics represent a drift in the on-device machine learning model over time.
15 . The system of claim 11 , wherein the instructions further cause the data processing hardware to receive a trigger from the remote system, wherein the instructions cause the data processing hardware to generate the corresponding performance data and transmit the one or more performance metrics to the remote system in response to receiving the trigger.
16 . The system of claim 15 , wherein the trigger comprises a cloud messaging push notification.
17 . The system of claim 15 , wherein the trigger comprises logic for generating the corresponding performance data.
18 . The system of claim 15 , wherein the trigger comprises at least one of an indication of a time period over which the corresponding performance data is to be generated, an indication of how often the corresponding performance data is to be generated, or an indication of how often the corresponding performance data is to be aggregated and reported.
19 . The system of claim 11 , wherein, to generate, using the training embedding snapshot and the corresponding on-device embedding, the corresponding performance data, the instructions cause the data processing hardware to determine one or more differences between the training embedding snapshot and the corresponding on-device embedding.
20 . The system of claim 19 , wherein the one or more differences comprise a cosine similarity or a Euclidean distance.