Granted Patent B1

US 12,216,702 · App. 15/008,956 · Granted Feb 4, 2025

Inventors: Kirk Ouimet (Orem, UT); Rong Yan (Marina Del Rey, CA); Yantao Zheng (Venice, CA); Timothy Michael Sehn (Marina Del Rey, CA); Evan Spiegel (Venice, CA)

Assignee: Snap Inc.

G06F16/51G06F16/583

Abstract

Claims (53)

1. A method to perform image-based searching, the method comprising:

for each of a plurality of visitor destinations at respective sub-locations of a real-world physical establishment:

receiving from a content owner associated with the establishment image data specific to the respective sub-location;

receiving from the content owner audio index data comprising a plurality of recordings made at the respective sub-location; and

receiving from the content owner location index data specific to the respective visitor destination;

in an automated procedure performed using one or more computer processor devices configured for performance of the automated procedure, compiling and storing on a machine-readable memory a respective index specific to each visitor destination, thereby provisioning a plurality of indexes corresponding to the plurality of visitor destinations, each index being provided by a respective data structure that is specific to the corresponding visitor destination and that comprises:

the image data of the respective visitor destination,

the location index data of the respective sub-location,

the audio index data associated with the respective sub-location, and

text index data that includes words that are frequently spoken by guides or visitors in proximity to the respective sub-location and that pertain to subject matter available for visitor consumption at the corresponding visitor destination;

receiving from a mobile computing device of a user a search query comprising one or more query items selected from: a query image, textual query data, and audio data associated with the query image;

in an automated operation based at least in part on the plurality of indexes and performed using one or more computer processor devices configured for performance of the automated operation, identifying as a target visitor destination at the establishment one of the plurality of visitor destinations at the respective sub-locations of the establishment based at least in part on determining a correlation between the search query and the index specific to the target visitor destination; and

responsive to determining of the correlation, based at least in part on the determining of the correlation and based at least in part on associated map information, directing the mobile computing device of the user to the sub-location of the identified target visitor destination situated within the establishment.

2. The method of claim 1 , wherein the determining of the correlation further comprises identifying a threshold degree of image similarity between the query image and the image data of the sub-location of the respective visitor destination used to create the index specific to the target visitor destination.

3. The method of claim 1 , wherein the image data of each sub-location of the plurality of visitor destination comprises a plurality of images that relate in common to the subject matter of the respective visitor destination.

4. The method of claim 1 , wherein content of the content owner that provides the respective image data comprises a publication of the content owner.

5. The method of claim 1 , wherein content of the content owner that provides the respective image data comprises a web page.

6. The method of claim 3 , wherein creating the index specific to the at least one of the plurality of visitor destinations comprises identifying the plurality of images as relating to the establishment associated with the content owner.

7. The method of claim 1 , wherein the establishment associated with the content owner is a museum, each of said plurality of visitor destinations being a respective display within the museum.

8. A system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, configure the system to:

for each of a plurality of visitor destinations at respective sub-locations of a real-world physical establishment:

receiving from a content owner associated with the establishment image data specific to the respective sub-location;

receiving from the content owner audio index data comprising a plurality of recordings made at the respective sub-location; and

receiving from the content owner location index data specific to the respective visitor destination;

in an automated procedure, compiling and storing on a machine-readable memory a respective index specific to each visitor destination, thereby provisioning a plurality of indexes corresponding to the plurality of visitor destinations, each index being provided by a respective data structure that is specific to the corresponding visitor destination and that comprises:

the image data of the respective visitor destination,

the location index data of the respective sub-location,

the audio index data associated with the respective sub-location, and

text index data that includes words that are frequently spoken by guides or visitors in proximity to the respective sub-location and that pertain to subject matter available for visitor consumption at the corresponding visitor destination;

receiving from a mobile computing device of a user a search query comprising one or more query items selected from: a query image, textual query data, and audio data associated with the query image;

in an automated operation based at least in part on the plurality of indexes, identifying as a target visitor destination at the establishment one of the plurality of visitor destinations at the respective sub-locations of the establishment based at least in part on determining a correlation between the search query and the index specific to the target visitor destination; and

responsive to the determining of the correlation, based at least in part on the determining of the correlation and based at least in part on associated map information, directing the mobile computing device of the user to the sub-location of the identified target visitor destination situated within the establishment.

9. The system of claim 8 , wherein the determining of the correlation further comprises identifying a threshold degree of image similarity between the query image and the image data of the sub-location of the respective visitor destination used to create the index specific to the target visitor destination.

10. The system of claim 8 , wherein the image data of each sub-location of the plurality of visitor destination comprises a plurality of images that relate in common to the subject matter of the respective visitor destination.

11. The system of claim 8 , wherein content of the content owner that provides the respective image data comprises a publication of the content owner.

12. The system of claim 10 , wherein creating the index specific to the at least one of the plurality of visitor destinations comprises identifying the plurality of images as relating to the establishment associated with the content owner.

13. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

for each of a plurality of visitor destinations at respective sub-locations of a real-world physical establishment:

receiving from a content owner associated with the establishment image data specific to the respective sub-location;

receiving from the content owner audio index data comprising a plurality of recordings made at the respective sub-location; and

receiving from the content owner location index data specific to the respective visitor destination;

in an automated procedure, compiling and storing on a machine-readable memory a respective index specific to each visitor destination, thereby provisioning a plurality of indexes corresponding to the plurality of visitor destinations, each index being provided by a respective data structure that is specific to the corresponding visitor destination and that comprises:

the image data of the respective visitor destination,

the location index data of the respective sub-location,

the audio index data associated with the respective sub-location, and

text index data that includes words that are frequently spoken by guides or visitors in proximity to the respective sub-location and that pertain to subject matter available for visitor consumption at the corresponding visitor destination;

receiving from a mobile computing device of a user a search query comprising one or more query items selected from: a query image, textual query data, and audio data associated with the query image;

in an automated operation based at least in part on the plurality of indexes, identifying as a target visitor destination at the establishment one of the plurality of visitor destinations at the respective sub-locations of the establishment based at least in part on determining a correlation between the search query and the index specific to the target visitor destination; and

responsive to the determining of the correlation, based at least in part on the determining of the correlation and based at least in part on associated map information, directing the mobile computing device of the user to the sub-location of the identified target visitor destination situated within the establishment.

14. The method of claim 2 , wherein the determining of the correlation is further based at least in part on identifying a degree of similarity between the audio index data and the audio data associated with the query image comprised in the search query.

15. The system of claim 9 , wherein the determining of the correlation is further based at least in part on identifying a degree of similarity between the audio index data and the audio data associated with the query image comprised in the search query.

Assignments (2)

Continuity (1)

References Cited (400)

Cited By (2)

US 12,411,890 US 12,645,736