Granted Patent B2

US 12,579,152 · App. 18/530,543 · Granted Mar 17, 2026

Inventors: Gökhan Hasan Bakir (Zurich, CH); Károly Csalogány (Zurich, CH); Behshad Behzadi (Zurich, CH)

Assignee: GOOGLE LLC

G06F16/24575G06F16/43G06F16/951G06F16/9538

Abstract

Claims (72)

1 . A method implemented by one or more processors, the method comprising:

identifying a plurality of multimedia content items;

extracting, for a given multimedia content item of the plurality of multimedia content items, one or more corresponding entities, wherein each of the one or more corresponding entities characterize one or more corresponding objects represented in the given multimedia content item, and wherein extracting the one or more corresponding entities for the given multimedia content item further comprises:

extracting one or more corresponding timestamps or time ranges, for each of the one or more corresponding entities, when which each of the one or more corresponding entities appear in the multimedia content, and

storing, in an index database and in association with each of the one or more corresponding entities, the one or more corresponding timestamps or time ranges when which each of the one or more corresponding entities appear in the multimedia content;

storing, in the index database, the one or more corresponding entities; and

subsequent to storing the one or more corresponding entities in the index database:

receiving input associated with a client computing device of a user, the input being associated with the given multimedia content item, and the input being received while the given multimedia content item is being streamed at the client computing device;

generating, based on the one or more corresponding entities for the given multimedia content item that are stored in the index database and based on the input, a query;

executing, based on the query, a search to obtain a result that is responsive to the query; and

causing the result to be provided for presentation to the user responsive to receiving the input and via the client computing device, the result being provided for presentation to the user without interrupting consumption of the given multimedia content item that is being streamed at the client computing device.

2 . The method of claim 1 , wherein extracting the one or more corresponding entities for the given multimedia content item further comprises:

extracting the one or more corresponding entities from metadata associated with the given multimedia content item.

3 . The method of claim 1 , wherein extracting the one or more corresponding entities for the given multimedia content item further comprises:

extracting the one or more corresponding entities from comments or annotations associated with the given multimedia content item.

4 . The method of claim 1 , further comprising:

prior to storing the one or more corresponding entities in the index database:

assigning a corresponding label to each of the one or more corresponding entities.

5 . The method of claim 1 , further comprising:

identifying, from a query repository, one or more historical queries associated with the given multimedia content item;

generating, based on the one or more corresponding entities for the given multimedia content item that are stored in the index database and based on the one or more historical queries, a plurality of rewrites of the one or more historical queries; and

storing, in the index database and in association with each of the one or more corresponding entities, one or more of the plurality of rewrites of the one or more historical queries.

6 . The method of claim 5 , further comprising:

ranking the plurality of rewrites of the one or more historical queries,

wherein storing one or more of the plurality of rewrites of the one or more historical queries comprises:

storing a highest ranking one of the plurality of rewrites of the one or more historical queries in the index database and in association with each of the one or more corresponding entities.

7 . The method of claim 6 , wherein ranking the plurality of rewrites of the one or more historical queries is based on corresponding results obtained responsive to submitting the plurality of rewrites of the one or more historical queries to a search engine.

8 . The method of claim 1 , wherein the one or more corresponding entities characterizing the at least one object represented in the given multimedia content item is one of: a name of a person represented in the given multimedia content item, a name of a person associated with the given multimedia content item, or a name of an object represented in the given multimedia content item.

9 . A system comprising:

memory comprising instructions; and

at least one processor configured to execute the instructions to:

identify a plurality of multimedia content items;

extract, for a given multimedia content item of the plurality of multimedia content items, one or more corresponding entities, wherein each of the one or more corresponding entities characterize one or more corresponding objects represented in the given multimedia content item, wherein, in extracting the one or more corresponding entities for the given multimedia content item, the at least one processor is further configured to execute the instructions to:

extract one or more corresponding timestamps or time ranges, for each of the one or more corresponding entities, when which each of the one or more corresponding entities appear in the multimedia content, and

store, in an index database and in association with each of the one or more corresponding entities, the one or more corresponding timestamps or time ranges when which each of the one or more corresponding entities appear in the multimedia content;

store, in the index database, the one or more corresponding entities; and

subsequent to storing the one or more corresponding entities in the index database:

receive input associated with a client computing device of a user, the input being associated with the given multimedia content item, and the input being received while the given multimedia content item is being streamed at the client computing device;

generate, based on the one or more corresponding entities for the given multimedia content item that are stored in the index database and based on the input, a query;

execute, based on the query, a search to obtain a result that is responsive to the query; and

cause the result to be provided for presentation to the user responsive to receiving the input and via the client computing device, the result being provided for presentation to the user without interrupting consumption of the given multimedia content item that is being streamed at the client computing device.

10 . The system of claim 9 , wherein, in extracting the one or more corresponding entities for the given multimedia content item, the at least one processor is further configured to execute the instructions to:

extract the one or more corresponding entities from metadata associated with the given multimedia content item.

11 . The system of claim 9 , wherein, in extracting the one or more corresponding entities for the given multimedia content item, the at least one processor is further configured to execute the instructions to:

extract the one or more corresponding entities from comments or annotations associated with the given multimedia content item.

12 . The system of claim 9 , wherein the at least one processor is further configured to execute the instructions to:

prior to storing the one or more corresponding entities in the index database:

assign a corresponding label to each of the one or more corresponding entities.

13 . The system of claim 9 , wherein the at least one processor is further configured to execute the instructions to:

identify, from a query repository, one or more historical queries associated with the given multimedia content item;

generate, based on the one or more corresponding entities for the given multimedia content item that are stored in the index database and based on the one or more historical queries, a plurality of rewrites of the one or more historical queries; and

store, in the index database and in association with each of the one or more corresponding entities, one or more of the plurality of rewrites of the one or more historical queries.

14 . The system of claim 13 , wherein the at least one processor is further configured to execute the instructions to:

rank the plurality of rewrites of the one or more historical queries,

wherein, in storing one or more of the plurality of rewrites of the one or more historical queries, the at least one processor is configured to execute the instructions to:

store a highest ranking one of the plurality of rewrites of the one or more historical queries in the index database and in association with each of the one or more corresponding entities.

15 . The system of claim 14 , wherein ranking the plurality of rewrites of the one or more historical queries is based on corresponding results obtained responsive to submitting the plurality of rewrites of the one or more historical queries to a search engine.

16 . The system of claim 9 , wherein the one or more corresponding entities characterizing the at least one object represented in the given multimedia content item is one of: a name of a person represented in the given multimedia content item, a name of a person associated with the given multimedia content item, or a name of an object represented in the given multimedia content item.

17 . The system of claim 9 , wherein, in causing the result to be provided for presentation to the user responsive to receiving the input and via the client computing device, the at least one processor is configured to execute the instructions to:

transmit data to the client computing device that causes the client computing device to provide the result for presentation to the user responsive to receiving the input; or

cause the client computing device to provide the result for presentation to the user responsive to receiving the input.

18 . A non-transitory machine-readable medium comprising instructions stored therein, which when executed by a processor, causes the processor to perform operations, the operations comprising:

identifying a plurality of multimedia content items;

extracting, for a given multimedia content item of the plurality of multimedia content items, one or more corresponding entities, wherein each of the one or more corresponding entities characterize one or more corresponding objects represented in the given multimedia content item, and wherein extracting the one or more corresponding entities for the given multimedia content item further comprises:

extracting one or more corresponding timestamps or time ranges, for each of the one or more corresponding entities, when which each of the one or more corresponding entities appear in the multimedia content, and

storing, in an index database and in association with each of the one or more corresponding entities, the one or more corresponding timestamps or time ranges when which each of the one or more corresponding entities appear in the multimedia content;

storing, in the index database, the one or more corresponding entities; and

subsequent to storing the one or more corresponding entities in the index database:

receiving input associated with a client computing device of a user, the input being associated with the given multimedia content item, and the input being received while the given multimedia content item is being streamed at the client computing device;

generating, based on the one or more corresponding entities for the given multimedia content item that are stored in the index database and based on the input, a query;

executing, based on the query, a search to obtain a result that is responsive to the query; and

causing the result to be provided for presentation to the user responsive to receiving the input and via the client computing device, the result being provided for presentation to the user without interrupting consumption of the given multimedia content item that is being streamed at the client computing device.

Assignments (2)

Continuity (4)

References Cited (114)