Server-assisted object recognition and tracking for mobile devices

1. A method, comprising:

capturing digital image data using a camera of a mobile device;

performing preliminary object detection on at least a portion of the image data to recognize an object and generate preliminary object detection data for the recognized object;

tracking the recognized object using the preliminary object detection data, while transmitting the at least a portion of the image data to a remote server for confirmation of the recognized object and refinement of the preliminary object detection data;

receiving from the remote server object data for the recognized object in the transmitted image data, the object data including data for an object reference frame for rendering on a display of the mobile device; and

refining tracking of the recognized object based on the received object data and the preliminary object detection data, wherein the tracking commences on the mobile device using the preliminary object detection data and continues after receiving the object data from the remote server using both the preliminary object detection data and the received object data.

2. The method of claim 1 , further comprising repeating the acts of capturing, transmitting, receiving, and tracking.

3. The method of claim 1 , wherein the preliminary object detection comprises a coarse-grained detection of text objects in the image data or a speculative detection of areas of interest in the image data.

4. The method of claim 1 , further comprising refining the preliminary object detection based at least in part on the received object data.

5. The method of claim 4 , wherein the refining the preliminary object detection involves at least one of:

eliminating one or more objects recognized through the preliminary object detection; and

receiving a translated text object associated with text in the preliminary object detection data.

6. The method of claim 1 , further comprising:

capturing one or more additional instances of digital image data using the camera; and

tracking changes in a location of the recognized object in the one or more additional instances of digital image data, the tracking being performed based at least in part on the object data received from the remote server.

7. The method of claim 6 , wherein the tracking further comprises updating the data for the object reference frame for the one or more additional instances of digital image data; and

wherein the method further comprises displaying the one or more additional instances of digital image data on the display of the mobile electronic device together with the updated object reference frame for the recognized object.

8. The method of claim 1 , wherein the transmitting and the receiving reduces a computational burden on the mobile device relative to performing object recognition only on the mobile device itself.

9. The method of claim 1 , further comprising transmitting sensor data to the remote server in addition to the at least a portion of the image data.

10. The method of claim 9 , wherein the sensor data further comprises one or more of data from an accelerometer associated with the mobile device, data from a compass associated with the mobile device, data from a gyroscope associated with the mobile device, or data associated with global positioning system coordinates for the mobile device.

11. The method of claim 9 , wherein the object data comprises data predicting a location of the recognized object based at least in part on the transmitted sensor data.

12. One or more computer-readable storage media storing computer-executable instructions which when executed by a computing device cause the computing device to perform a method for processing image data, the stored instructions comprising:

instructions to capture digital image data using a camera of a mobile device;

instructions to perform preliminary object detection on at least a portion of the image data to recognize an object and generate preliminary object detection data for the recognized object;

instructions to track the recognized object using the preliminary object detection data, while transmitting the at least a portion of the image data to a remote server for confirmation of the recognized object and refinement of the preliminary object detection data;

instructions to receive from the remote server object data for the recognized object in the transmitted image data, the object data including data for an object reference frame for rendering on a display of the mobile device; and

instructions to refine tracking of the recognized object based on the received object data and the preliminary object detection data, wherein the tracking commences on the mobile device using the preliminary object detection data and continues after receiving the object data from the remote server using both the preliminary object detection data and the received object data.

13. A system, comprising:

a memory or storage device storing a program; and

at least one processor that executes the program, wherein said execution of the program causes the at least one processor to:

capture digital image data using a camera of a mobile device;

perform preliminary object detection on at least a portion of the image data to recognize an object and generate preliminary object detection data for the recognized object;

track the recognized object using the preliminary object detection data, while transmitting the at least a portion of the image data to a remote server for confirmation of the recognized object and refinement of the preliminary object detection data;

receive from the remote server object data for the recognized object in the transmitted image data, the object data including data for an object reference frame for rendering on a display of the mobile device; and

refine tracking of the recognized object based on the received object data and the preliminary object detection data, wherein the tracking commences on the mobile device using the preliminary object detection data and continues after receiving the object data from the remote server using both the preliminary object detection data and the received object data.

14. The system of claim 13 , wherein said execution of the program causes the at least one processor to:

repeat the acts of capturing, transmitting, receiving, and tracking.

15. The system of claim 13 , wherein the preliminary object detection comprises a coarse-grained detection of text objects in the image data or a speculative detection of areas of interest in the image data.

16. The system of claim 13 , wherein said execution of the program causes the at least one processor to refine the preliminary object detection based at least in part on the received object data, wherein the refining the preliminary object detection involves at least one of:

eliminating one or more objects recognized through the preliminary object detection; and

receiving a translated text object associated with text in the preliminary object detection data.

17. The system of claim 13 , wherein said execution of the program causes the at least one processor to:

capture one or more additional instances of digital image data using the camera; and

track changes in a location of the recognized object in the one or more additional instances of digital image data, the tracking being performed based at least in part on the object data received from the remote server.

18. The system of claim 17 , wherein the tracking further causes the at least one processor to update the data for the object reference frame for the one or more additional instances of digital image data; and

wherein said execution of the program causes the processing unit to displaying the one or more additional instances of digital image data on the display of the mobile electronic device together with the updated object reference frame for the recognized object.

19. The system of claim 13 , wherein the transmitting and the receiving reduces a computational burden on the mobile device relative to performing object recognition only on the mobile device itself.

20. The system of claim 13 , wherein said execution of the program causes the at least one processor to transmit sensor data to the remote server in addition to the at least a portion of the image data;

wherein the sensor data further comprises one or more of data from an accelerometer associated with the mobile device, data from a compass associated with the mobile device, data from a gyroscope associated with the mobile device, or data associated with global positioning system coordinates for the mobile device; and

wherein the object data comprises data predicting a location of the recognized object based at least in part on the transmitted sensor data.