SYSTEM AND METHOD FOR EDGE INFERENCE
Systems and methods for selectively applying inference models on one or more edge devices in response to actual or predicted delays are disclosed. Inference models may be trained and deployed to a server and a first edge device. Sensor data may be received at the server and may also be forwarded to the first edge device. A first inference may be performed on the server by applying the data to the trained inference model to generate a first inference result. The results may be sent to the first edge device. In response to not receiving the first inference result at the first edge device after a delay threshold or in response to a queue length on the server exceeding a threshold, an inference may be performed on the first edge device using the received sensor data. Inference results from the server and the edge device may be combined and reordered.
1 . A method for processing data, the method comprising:
(a) training an inference model;
(b) deploying the inference model to a server and a first edge device;
(c) receiving sensor data from a second edge device at the server and at the first edge device;
(d) performing a first inference on the server by applying the sensor data to the inference model to generate a first inference result;
(e) sending the first inference result to the first edge device; and
(f) performing a second inference on the sensor data on the first edge device in response to not receiving the first inference result at the first edge device after a predetermined delay threshold.
2 . The method of claim 1 , wherein the sensor data is audio data.
3 . The method of claim 2 , wherein the first inference and the second inference comprises inferring a text translation based on the audio data.
4 . The method of claim 1 , wherein the first inference and the second inference comprises inferring a stress level based on the sensor data.
5 . The method of claim 1 , wherein the sensor data is image data or video data.
6 . The method of claim 1 , further comprising aborting the second inference if the first inference result is received by the first edge device prior to the second inference being completed.
7 . A method for processing data, the method comprising:
(a) receiving a first set of data from a first device in a queue for processing on a server;
(b) performing a first inference on the first set of data to generate a first result using a trained inference model in response to the queue being shorter than a predetermined threshold; and
(c) in response to the queue being longer than the predetermined threshold:
(i) sending the first set of data to a second device,
(ii) instructing the second device to perform a second inference on the first set of data to generate a second result.
8 . The method of claim 7 , wherein (c) further comprises deploying the trained inference model to the second device.
9 . The method of claim 7 , wherein the second device is an edge device or a mobile phone.
10 . The method of claim 7 , wherein the first set of data is audio data, image data, or video data.
11 . The method of claim 10 , wherein the first inference and the second inference comprise inferring a stress level based on the first set of data.
12 . The method of claim 7 , wherein the first inference and the second inference comprises inferring a text translation based on the first set of data.
13 . The method of claim 7 , further comprising caching the first set of data on the second device.
14 . A method for processing data, the method comprising:
(a) training an inference model;
(b) deploying the inference model to a first computer;
(c) creating a first queue on the first computer;
(d) receiving a first set of data from a first device in the first queue;
(e) predicting a wait time for the first queue;
(f) applying the first set of data to the inference model on the first computer and sending results to a second device; and
(g) in response to the predicted wait time for the first queue being greater than a predetermined threshold:
(i) deploying the inference model to the second device;
(ii) creating a second queue on the second device; and
(iii) directing at least a portion of subsequent sets of data to the second queue in lieu of the first queue.
15 . The method of claim 14 , further comprising:
(h) generating a first stream of inference results on the first computer;
(i) forwarding the first stream of inference results to the second device;
(j) generating a second stream of inference results on the second device; and
(k) ordering the first stream of inference results and the second stream of inference results on the second device.
16 . The method of claim 14 , wherein the first set of data includes audio data, and wherein the inference model infers a text translation based on the audio data.
17 . The method of claim 14 , wherein the data is image data or video data.
18 . A method for processing data, the method comprising:
(a) training an inference model;
(b) deploying the inference model to a first computer;
(c) creating a first queue on the first computer;
(d) receiving a first set of data captured by a sensor on a first device in the first queue;
(e) performing a first inference on the first computer to generate a first result;
(f) sending the first result to a client;
(g) predicting a wait time for the first queue; and
(h) in response to the predicted wait time being greater than a predetermined threshold:
(i) deploying the inference model to the first device, and
(ii) instructing the first device to apply the inference model to at least a subset of subsequent data captured by the sensor and send subsequent results to the client.
19 . The method of claim 18 , wherein (h) further comprises:
(iii) ordering the first stream of inference results from the first computer and the subsequent results from the first device at the client based on a sequence id.
20 . The method of claim 19 , wherein the sequence id is a timestamp.