Optimizing image offloading in edge-assisted augmented reality
An offloading schedule for offloading image frames that are to be generated by an AR device in a subsequent time period is determined, the offloading schedule identifying certain image frames generated in the subsequent time period that are to be offloaded and certain image frames generated in the subsequent time period that are not to be offloaded, the offloading schedule being selected from a plurality of offloading schedules based on a tracking stride of the offloading schedule. The AR device sends to a computing device at least some of the image frames generated in the subsequent time period in accordance with the offloading schedule.
1. A method comprising:
determining, by an augmented reality (AR) device, a first offloading schedule for offloading image frames that are to be generated by the AR device in a first subsequent time period, the first offloading schedule identifying certain image frames generated in the first subsequent time period that are to be offloaded and certain image frames generated in the first subsequent time period that are not to be offloaded, the first offloading schedule being selected from a first plurality of offloading schedules based on a tracking stride of the first offloading schedule; and
sending, by the AR device to a computing device, at least some of the image frames generated in the first subsequent time period in accordance with the first offloading schedule.
2. The method of claim 1 wherein the tracking stride identifies a frame difference between a most recently generated image frame and a most recent previous image frame for which the AR device has received image processing results from the computing device.
3. The method of claim 1 further comprising:
determining, by the AR device, the first offloading schedule based on an uplink transfer time (UTT) for an image frame to be communicated from the AR device to the computing device, a downlink transfer time (DTT) for image processing results generated by the computing device based on the image frame to be communicated from the computing device to the AR device, an image processing time for the computing device to process the image frame to generate the image processing results, and a camera frame rate of the AR device.
4. The method of claim 3 further comprising:
sending, by the AR device to the computing device, a first image frame for image processing;
storing, by the AR device, a send timestamp that identifies a time at which the first image frame was sent to the computing device;
receiving, by the AR device, image processing results from the computing device based on the first image frame, the image processing results comprising a start processing timestamp that identifies a start time of image processing by the computing device, a finish processing timestamp that identifies a finish time of the image processing by the computing device;
determining, by the AR device, a receive timestamp that identifies a time that the image processing results were received by the AR device; and
determining the UTT, the DTT, and the image processing time of the first image frame based on the send timestamp, the start processing timestamp, the finish processing timestamp, and the receive timestamp.
5. The method of claim 4 further comprising:
determining, by the AR device, the UTT, the DTT, and the image processing time of the first image frame periodically during a continuous AR session.
6. The method of claim 4 further comprising:
determining, by the AR device, the UTT, the DTT, and the image processing time each time the AR device sends an image frame to the computing device for image processing.
7. The method of claim 4 further comprising:
determining, by the AR device, the UTT, the DTT, and the image processing time based on corresponding moving window averages of consecutively determined UTTs, DTTs, and image processing times.
8. The method of claim 1 further comprising:
wherein determining the first offloading schedule further comprises:
determining, based on an uplink transfer time (UTT) for an image frame to be communicated from the AR device to the computing device, a downlink transfer time (DTT) for image processing results generated by the computing device based on the image frame to be communicated from the computing device to the AR device, an image processing time for the computing device to process the image frame to generate the image processing results, and a camera frame rate of the AR device, a plurality of offloading schedules; and
selecting the first offloading schedule from the plurality of offloading schedules based on the tracking stride of the first offloading schedule.
9. The method of claim 8 wherein selecting the first offloading schedule from the plurality of offloading schedules based on the tracking stride of the first offloading schedule further comprises:
analyzing a subset of the plurality of offloading schedules;
determining a tracking stride of each offloading schedule in the subset of the plurality of offloading schedules; and
selecting the first offloading schedule in response to the first offloading schedule having a lowest tracking stride of each offloading schedule in the subset of the plurality of offloading schedules.
10. The method of claim 9 wherein the plurality of offloading schedules is greater than 250,000 offloading schedules.
11. The method of claim 9 wherein analyzing the subset of the plurality of offloading schedules comprises randomly selecting the subset of the plurality of offloading schedules from the plurality of offloading schedules.
12. The method of claim 8 wherein T-inf equals the image processing time of the computing device divided by an image frame interval of the AR device, and wherein prior to selecting the first offloading schedule from the first plurality of offloading schedules based on the tracking stride of the first offloading schedule, pruning the first plurality of offloading schedules to remove each offloading schedule that offloads two image frames that are less than T-inf frames apart at any point in the offloading schedule, and pruning the first plurality of offloading schedules to remove each offloading schedule that offloads two image frames that are more than T-inf frames apart at any point in the offloading schedule.
13. The method of claim 8 further comprising:
generating, by the AR device based on the image processing results, AR imagery comprising real-world imagery depicting a real-world scene within a field of view of a camera of the AR device, and computer-generated imagery depicting an object not in the real-world scene.
14. The method of claim 1 wherein sending, by the AR device to the computing device, the at least some of the image frames generated in the first subsequent time period in accordance with the first offloading schedule comprises sending, by the AR device to the computing device, the at least some of the image frames generated in the first subsequent time period in accordance with the first offloading schedule via a cellular network.
15. The method of claim 1 further comprising:
prior to generation of each image frame in the first subsequent time period, determining, by the AR device, a second offloading schedule for offloading image frames that are to be generated by the AR device in a second subsequent time period, the second subsequent time period overlapping the first subsequent time period, the second offloading schedule identifying certain image frames generated in the second subsequent time period that are to be offloaded and certain image frames generated in the second subsequent time period that are not to be offloaded, the second offloading schedule being selected from a second plurality of offloading schedules based on a tracking stride of the second offloading schedule; and
sending, by the AR device to the computing device, at least some of the image frames generated in the second subsequent time period in accordance with the second offloading schedule.
16. An augmented reality (AR) device, comprising:
a camera;
a memory; and
a processor device coupled to the memory and the camera and being operable to:
determine a first offloading schedule for offloading image frames that are to be generated by the AR device in a first subsequent time period, the first offloading schedule identifying certain image frames generated in the first subsequent time period that are to be offloaded and certain image frames generated in the first subsequent time period that are not to be offloaded, the first offloading schedule being selected from a first plurality of offloading schedules based on a tracking stride of the first offloading schedule; and
send, to a computing device, at least some of the image frames generated in the first subsequent time period in accordance with the first offloading schedule.
17. The AR device of claim 16 wherein the tracking stride identifies a frame difference between a most recently generated image frame and a most recent previous earlier image frame for which the AR device has received image processing results from the computing device.
18. The AR device of claim 16 wherein the processor device is further operable to:
determine the first offloading schedule based on an uplink transfer time (UTT) for an image frame to be communicated from the AR device to the computing device, a downlink transfer time (DTT) for image processing results generated by the computing device based on the image frame to be communicated from the computing device to the AR device, an image processing time for the computing device to process the image frame to generate the image processing results, and a camera frame rate of the AR device.
19. The AR device of claim 16 wherein to determine the first offloading schedule, the processor device is further operable to:
determine, based on an uplink transfer time (UTT) for an image frame to be communicated from the AR device to the computing device, a downlink transfer time (DTT) for image processing results generated by the computing device based on the image frame to be communicated from the computing device to the AR device, an image processing time for the computing device to process the image frame to generate the image processing results, and a camera frame rate of the AR device, a plurality of offloading schedules; and
select the first offloading schedule from the plurality of offloading schedules based on the tracking stride of the first offloading schedule.
20. A non-transitory computer-readable storage medium that includes executable instructions configured to cause a processor device of an augmented reality (AR) device to:
determine a first offloading schedule for offloading image frames that are to be generated by the AR device in a first subsequent time period, the first offloading schedule identifying certain image frames generated in the first subsequent time period that are to be offloaded and certain image frames generated in the first subsequent time period that are not to be offloaded, the first offloading schedule being selected from a first plurality of offloading schedules based on a tracking stride of the first offloading schedule; and
send, to a computing device, at least some of the image frames generated in the first subsequent time period in accordance with the first offloading schedule.