Granted Patent B2

US 12,651,418 · App. 18/317,851 · Granted Jun 9, 2026

Inventors: Jennifer Healey (San Jose, CA); Tong Sun (San Ramon, CA); Nicholas Rewkowski (San Jose, CA); Nedim Lipka (Campbell, CA); Curtis Wigington (San Jose, CA); Alexa Siu (San Jose, CA)

Assignee: Adobe Inc.

G06T19/006G10L15/08G10L15/22G06T2219/004G10L2015/088

Abstract

Claims (94)

1 . A method comprising:

capturing image data using a camera;

capturing spatial information using one or more sensors;

receiving voice data using a microphone;

analyzing the voice data to identify a keyword;

analyzing the image data and the spatial information to identify an object corresponding to the keyword;

generating text based on the voice data and the keyword;

storing the text in association with the object; and

generating and providing output comprising the text linked to the object or a derivative thereof, wherein generating the output comprises:

capturing second image data;

identifying one or more image markers in the second image data;

traversing stored image data to identify the image markers;

retrieving the generated text based on an association with the stored image data;

retrieving data identifying the object in the stored image data;

aligning a second object in the second image data with the object in the stored image data; and

overlaying the text on the object based on the alignment.

2 . The method of claim 1 , wherein the method is performed by a computing device and storing the text in association with the object comprises:

storing an anchor location with respect to a position of the computing device;

storing information characterizing an intended viewpoint; and

storing the text in association with the anchor location and the information characterizing the intended viewpoint.

3 . The method of claim 1 , further comprising:

establishing a set of weights, each weight corresponding to a type of image, audio, or sensor data based on an accuracy thereof; and

computing a saliency value as a function of input data and respective weights of the set of weights, wherein the weights and the saliency value are further stored in association with the object and the text.

4 . The method of claim 1 , wherein the text is a first text, the method further comprising:

receiving a second text;

identifying a relationship between the second text and the first text or the object; and

updating the stored text in association with the object to comprise the second text and an indication of the relationship.

5 . The method of claim 4 , wherein the first text is received from a first user and the second text is received from a second user.

6 . The method of claim 1 , further comprising:

based on the image data, the spatial information, and the voice data, identifying and tracking one or more floors in a building,

wherein the spatial information comprises GPS data and gyroscope data.

7 . The method of claim 1 , further comprising storing, with the text in association with the object:

metadata specifying a date and a user.

8 . A system comprising:

a memory component; and

a processing device coupled to the memory component, the processing device to perform operations comprising:

capturing image data using a camera;

capturing spatial information using one or more sensors;

receiving user input associated with the image data;

analyzing the user input to identify a keyword;

analyzing the image data and the spatial information to identify an object corresponding to the keyword;

generating text based on the user input and the keyword;

storing the text in association with the object; and

generating and providing output comprising the text linked to the object or a derivative thereof, wherein generating the output comprises:

capturing second image data;

identifying one or more image markers in the second image data;

traversing stored image data to identify the image markers;

retrieving the generated text based on an association with the stored image data;

retrieving data identifying the object in the stored image data;

aligning a second object in the second image data with the object in the stored image data; and

overlaying the text on the object based on the alignment.

9 . The system of claim 8 , wherein the operations are performed by a computing device, the operations further comprising:

storing an anchor location with respect to a position of the computing device;

storing information characterizing an intended viewpoint; and

storing the text in association with the anchor location and the information characterizing the intended viewpoint.

10 . The system of claim 8 , the operations further comprising:

establishing a set of weights, each weight corresponding to a type of image, audio, or sensor data based on an accuracy thereof; and

computing a saliency value as a function of input data and respective weights of the set of weights, wherein the weights and the saliency value are further stored in association with the object and the text.

11 . The system of claim 8 , wherein the text is a first text, the operations further comprising:

receiving a second text;

identifying a relationship between the second text and the first text or the object; and

updating the stored text in association with the object to comprise the second text and an indication of the relationship.

12 . The system of claim 11 , wherein:

the first text is received from a first user and the second text is received from a second user.

13 . The system of claim 8 , the operations further comprising:

based on the image data, the spatial information, and the user input, identifying and tracking one or more floors in a building,

wherein the spatial information comprises GPS data and gyroscope data.

14 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

capturing image data using a camera;

capturing spatial information using one or more sensors;

receiving voice data using a microphone;

analyzing the voice data to identify a keyword;

analyzing the image data and the spatial information to identify an object corresponding to the keyword;

generating text based on the voice data and the keyword;

storing the text in association with the object; and

generating and providing output comprising the text linked to the object or a derivative thereof, wherein generating the output comprises:

capturing second image data;

identifying one or more image markers in the second image data;

traversing stored image data to identify the image markers;

retrieving the generated text based on an association with the stored image data;

retrieving data identifying the object in the stored image data;

aligning a second object in the second image data with the object in the stored image data; and

overlaying the text on the object based on the alignment.

15 . The medium of claim 14 , wherein the operations are performed by a computing device, the operations further comprising:

storing an anchor location with respect to a position of the computing device;

storing information characterizing an intended viewpoint; and

storing the text in association with the anchor location and the information characterizing the intended viewpoint.

16 . The medium of claim 14 , the operations further comprising:

establishing a set of weights, each weight corresponding to a type of image, audio, or sensor data based on an accuracy thereof; and

computing a saliency value as a function of input data and respective weights of the set of weights, wherein the weights and the saliency value are further stored in association with the object and the text.

17 . The medium of claim 14 , wherein the text is a first text received from a first user, the operations further comprising:

receiving a second text from a second user;

identifying a relationship between the second text and the first text or the object; and

updating the stored text in association with the object to comprise the second text and an indication of the relationship.

Assignments (1)

Continuity (1)

References Cited (48)