Granted Patent B2

US 12,645,736 · App. 19/002,438 · Granted Jun 2, 2026

Inventors: Kirk Ouimet (Orem, UT); Rong Yan (Marina Del Rey, CA); Yantao Zheng (Sunnyvale, CA); Timothy Michael Sehn (Marina Del Rey, CA); Evan Spiegel (Los Angeles, CA)

Assignee: Snap Inc.

G06F16/78G06F16/434G06F16/487G06F16/489G06F16/51G06F16/58G06F16/583G06F16/587G11B27/11G11B27/34G06F3/167

Abstract

Claims (78)

1 . A method comprising:

for each of a plurality of sub-locations of a geographic location, compiling and storing on a machine-readable memory a respective sub-location index, each sub-location index comprising a set of content items comprising:

image data comprising photographic and/or video items captured at the respective sub-location, and

text data comprising verbiage frequently spoken at the respective sub-location;

receiving from a mobile computing device associated with a user a search query comprising image query data; and

in an automated operation performed using one or more computer processor devices, performing operations comprising:

performing image analysis of the image query data;

based on the image analysis, compiling an augmented search query by generating and incorporating in the search query a set of keywords;

identifying as search result a specific sub-location based at least in part on determining a correlation between the augmented search query and the respective sub-location index of said specific sub-location;

returning to the mobile computing device responsive to the search query the search result identifying said specific sub-location; and

directing the mobile computing device to the specific sub-location indicated by the search result.

2 . The method of claim 1 , wherein the text data of each sub-location index comprises a textual description of the respective sub-location.

3 . The method of claim 1 , wherein the performing of the image analysis comprises:

performing object recognition to identify multiple distinct objects within the image query data;

generating a respective keyword for each identified object; and

incorporating the generated keywords into the augmented search query.

4 . The method of claim 3 , wherein the object recognition comprises:

parsing the image query data to identify different components and aspects thereof; and

generating descriptive keywords for each identified component and aspect.

5 . The method of claim 1 , wherein compiling the augmented search query comprises:

providing the image query data to a third-party provider;

receiving keywords from the third-party provider based on their analysis of the image query data; and

incorporating the received keywords into the augmented search query.

6 . The method of claim 1 , further comprising:

receiving audio data associated with the image query data;

performing voice recognition on the received audio data to generate textual data; and

incorporating the generated textual data into the augmented search query.

7 . The method of claim 6 , wherein the performing of the voice recognition comprises:

filtering the received audio data using at least one of a band-pass filter and a high-pass filter to isolate voice frequencies; and

generating text from the filtered audio data.

8 . The method of claim 1 , further comprising:

receiving ambient audio data captured at a location of the mobile computing device;

processing the ambient audio data through an audio fingerprint component to generate location-identifying text; and

incorporating the location-identifying text into the augmented search query.

9 . The method of claim 8 , wherein the processing of the ambient audio data comprises:

filtering the ambient audio data using a low-pass filter to isolate background audio data;

comparing the filtered background audio data to stored audio fingerprints; and

generating text metadata based on matching audio fingerprints.

10 . The method of claim 1 , wherein the compiling of the augmented search query comprises:

analyzing received audio data to identify spoken descriptions of content captured in the image query data; and

incorporating text corresponding to the identified spoken descriptions into the augmented search query.

11 . The method of claim 1 , wherein the geographic location is a museum, each sub-location comprising a respective display within the museum.

12 . A system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, configure the system to:

for each of a plurality of sub-locations of a geographic location, compile and store on a machine-readable memory a respective sub-location index, each sub-location index comprising a set of content items comprising:

image data comprising photographic and/or video items captured at the respective sub-location, and

text data comprising a textual description of the respective sub-location;

receive from a mobile computing device associated with a user a search query comprising image query data;

perform image analysis of the image query data;

based on the image analysis, compile an augmented search query by generating and incorporating in the search query a set of keywords;

identify as search result a specific sub-location based at least in part on determining a correlation between the augmented search query and the respective sub-location index of said specific sub-location;

return to the mobile computing device responsive to the search query the search result identifying said specific sub-location; and

direct the mobile computing device to the specific sub-location indicated by the search result.

13 . The system of claim 12 , wherein the text data of each sub-location index comprises verbiage frequently spoken at the respective sub-location.

14 . The system of claim 12 , wherein the performing of the image analysis comprises:

performing object recognition to identify multiple distinct objects within the image query data;

generating a respective keyword for each identified object; and

incorporating the generated keywords into the augmented search query.

15 . The system of claim 12 , wherein the instructions further configure the system to:

receive audio data associated with the image query data;

perform voice recognition on the received audio data to generate textual data; and

incorporate the generated textual data into the augmented search query.

16 . The system of claim 12 , wherein the instructions further configure the system to:

receive ambient audio data captured at a location of the mobile computing device;

process the ambient audio data through an audio fingerprint component to generate location-identifying text; and

incorporate the location-identifying text into the augmented search query.

17 . The system of claim 12 , wherein the geographic location is a museum, each sub-location comprising a respective display within the museum.

18 . A non-transitory computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

for each of a plurality of sub-locations of a geographic location, compile and store on a machine-readable memory a respective sub-location index, each sub-location index comprising a set of content items comprising:

image data comprising photographic and/or video items captured at the respective sub-location, and

text data comprising verbiage frequently spoken at the respective sub-location;

receive from a mobile computing device associated with a user a search query comprising image query data;

perform image analysis of the image query data;

based on the image analysis, compile an augmented search query by generating and incorporating in the search query a set of keywords;

identify as search result a specific sub-location based at least in part on determining a correlation between the augmented search query and the respective sub-location index of said specific sub-location;

return to the mobile computing device responsive to the search query the search result identifying said specific sub-location; and

direct the mobile computing device to the specific sub-location indicated by the search result.

Assignments (1)

Continuity (3)

References Cited (400)