Systems and methods for edge assisted real-time object detection for mobile augmented reality
View Patent ↗Systems and methods for edge assisted real-time object detection for mobile augmented reality are provided. The system employs a low latency offloading process, decouples the rendering pipeline from the offloading pipeline, and uses a fast object tracking method to maintain detection accuracy. The system can operate on a mobile device, such as an AR device, and dynamically offloads computationally-intensive object detection functions to an edge cloud device using an adaptive offloading process. The system also includes dynamic RoI encoding and motion vector-based object tracking processes that operate in a tracking and rendering pipeline executing on the AR device.
1. A system for edge assisted real-time object detection for mobile augmented reality, comprising:
an AR device having a processor, the processor of the AR device capturing at least one image frame and dynamically encoding a region of interest in the at least one image frame; and
an edge cloud computing device in communication with the AR device,
wherein the AR device adaptively offloads the region of interest to the edge cloud computing device, and wherein the edge cloud computing device performs image recognition in the region of interest and transmits results of the image recognition to the AR device, and
wherein the AR device processes the at least one image frame using a motion vector-based object tracking process.
2. The system of claim 1 , wherein the edge cloud computing device detects an object in the region of interest and transmits the detected object to the AR device, and the AR device renders the detected object and the at least one image frame.
3. The system of claim 1 , wherein the AR device transmits the at least one image frame to the edge cloud device in portions, the edge cloud computing device processing the portions.
4. The system of claim 3 , wherein the AR device streams the portions of the at least one image in parallel to the edge cloud computing device.
5. The system of claim 3 , wherein the edge cloud computing device processes the portions of the at least one image in parallel.
6. The system of claim 1 , wherein the edge cloud device detects the object in the region of interest using a neural network.
7. The system of claim 1 , wherein the AR device stores the detected object in a cache.
8. The system of claim 1 , wherein the AR device compresses at least part of the region of interest prior to offloading the region of interest to the edge cloud computing device.
9. The system of claim 1 , wherein dynamic encoding of the region of interest by the AR device reduces transmission latency and bandwidth consumption when the region of interest is offloaded to the edge cloud computing device.
10. A method for edge assisted real-time object detection for mobile augmented reality, comprising the steps of:
capturing at least one image frame at an AR device;
dynamically encoding a region of interest in the at least one image frame at the AR device;
adaptively offloading the region of interest to an edge cloud computing device;
performing image recognition in the region of interest at the edge cloud computing device;
transmitting results of the image recognition to the AR device; and
processing the at least one image frame at the AR device using a motion vector-based object tracking process.
11. The method of claim 10 , further comprising detecting an object in the region of interest, transmitting the detected object to the AR device, and rendering the detected object and the at least one image frame.
12. The method of claim 10 , further comprising transmitting the at least one image frame from the AR device to the edge cloud device in portions, the edge cloud device processing the portions of the at least one image.
13. The method of claim 12 , further comprising streaming the portions of the at least one image frame from the AR device to the edge cloud device in parallel.
14. The method of claim 12 , further comprising processing the portions of the at least one image frame at the edge cloud device in parallel.
15. The method of claim 10 , further comprising detecting the object in the region of interest using a neural network.
16. The method of claim 10 , further comprising storing the detected object in a cache in the AR device.
17. The method of claim 10 , further comprising compressing at least part of the region of interest prior to offloading the region of interest to the edge cloud computing device.
18. The method of claim 10 , wherein the step of dynamically encoding of the region of interest reduces transmission latency and bandwidth consumption when the region of interest is offloaded to the edge cloud computing device.
19. A method for edge assisted real-time object detection for mobile augmented reality, comprising the steps of:
capturing at least one image frame at an AR device;
transmitting the at least one image frame to an edge cloud computing device in portions sent in parallel to the edge cloud computing device;
performing image recognition on the at least one image frame at the edge cloud computing device;
transmitting results of the image recognition to the AR device; and
processing the at least one image frame at the AR device using a motion vector-based object tracking process.
20. The method of claim 19 , further comprising detecting an object in at least one image frame, transmitting the detected object to the AR device, and rendering the detected object and the at least one image frame.
21. The method of claim 19 , further comprising detecting the object using a neural network.
22. The method of claim 19 , further comprising storing the detected object in a cache in the AR device.
23. The method of claim 19 , further comprising compressing at least a portion of the at least one image frame prior to transmitting the at least one image frame to the edge cloud computing device.