Visual menu
An augmented reality (AR) overlay augments traditional menu items with corresponding photos, thereby facilitating a decision-making process of a user ordering from the menu. In addition to providing imagery of the menu items listed, other information may also be supplied, such as ratings, reviews etc. In this regard, users can visualize what to expect before ordering, and can order with a greater degree of confidence that they will enjoy the menu item they select.
1. A method of identifying images corresponding to text items, comprising:
receiving, with one or more processors, captured image of captured text from an image capture device;
parsing, with the one or more processors, the captured text in the captured image;
determining, with the one or more processors, a location of the image capture device at a time the captured image was captured;
determining, with the one or more processors, an entity corresponding to the determined location;
identifying images corresponding to the parsed text and the entity;
selecting at least one of the identified images by generating a first set of labels for the captured image based on the parsed text, generating a second set of labels for the identified images corresponding to the parsed text and the entity, and comparing the first set of labels to the second set of labels; and
providing the selected image for display as an augmented reality overlay in relation to the captured text.
2. The method of claim 1 , wherein comparing the first set of labels to the second set of labels comprises determining a distance between the first set of labels and the second set of labels.
3. The method of claim 2 , wherein selecting the at least one of the identified images comprises identifying a shortest distance between the first set of labels and the second set of labels, and selecting an image of the at least one of the identified images corresponding to the shortest distance.
4. The method of claim 1 , further comprising:
generating a score for each of the identified images; and
ranking the identified images based on the scores.
5. The method of claim 4 , wherein each score is at least partially based on image quality and image aesthetics.
6. The method of claim 1 , wherein providing the selected image for display comprises attaching the selected image to the captured text.
7. The method of claim 1 , wherein the captured text is a menu item, the entity is a restaurant, and the identified images are images of a dish served at the restaurant corresponding to the menu item.
8. The method of claim 1 , wherein parsing the captured text comprises optical character recognition.
9. The method of claim 1 , wherein identifying the images corresponding to the parsed text and the entity comprises retrieving images from one or more websites associated with the entity.
10. A system for identifying images corresponding to text items, comprising:
one or more memories;
one or more processors in communication with the one or more memories, the one or more processors configured to:
receive a captured image of captured text from an image capture device;
parse the captured text in the captured image;
determine a location of the image capture device at a time the captured image was captured;
determine a entity corresponding to the determined location;
identify images corresponding to the parsed text and the entity;
select at least one of the identified images by generating a first set of labels for the captured image based on the parsed text, generating a second set of labels for the identified images corresponding to the parsed text and the entity, and comparing the first set of labels to the second set of labels; and
provide the selected image for display as an augmented reality overlay in relation to the captured text.
11. The system of claim 10 , wherein comparing the first set of labels to the second set of labels comprises determining a distance between the first set of labels and the second set of labels.
12. The system of claim 11 , wherein selecting the at least one of the identified images comprises identifying a shortest distance between the first set of labels and the second set of labels, and selecting an image of the at least one of the identified images corresponding to the shortest distance.
13. The system of claim 10 , further comprising:
generating a score for each of the identified images; and
ranking the identified images based on the scores.
14. The system of claim 13 , wherein each score is at least partially based on image quality and image aesthetics.
15. The system of claim 10 , wherein providing the selected image for display comprises attaching the selected image to the captured text.
16. The system of claim 10 , wherein the captured text is a menu item, the entity is a restaurant, and the identified images are images of a dish served at the restaurant corresponding to the menu item.
17. The system of claim 10 , wherein the one or more processors reside on a client device.
18. A non-transitory computer-readable medium storing instructions executable by one or more processors for performing a method of identifying images corresponding to text items, the method comprising:
receiving a captured image of captured text from an image capture device;
parsing the captured text in the captured image;
determining a location of the image capture device at a time the captured image was captured;
determining a entity corresponding to the determined location;
identifying images corresponding to the parsed text and the entity;
selecting at least one of the identified images by generating a first set of labels for the captured image based on the parsed text, generating a second set of labels for the identified images corresponding to the parsed text and the entity, and comparing the first set of labels to the second set of labels; and
providing the selected image for display as an augmented reality overlay in relation to the captured text.