IP Library › Granted Patent US 12,411,542
Granted Patent B2
US 12,411,542 · App. 18/607,296 · Granted Sep 9, 2025

Electronic devices using object recognition and/or voice recognition to provide personal and health assistance to users

Inventors: Kenneth Luke Kocienda (Mill Valley, CA); Yanir Nulman (San Francisco, CA); Lilynaz Hashemi (San Francisco, CA); Imran A Chaudhri (San Francisco, CA); Jane Koo (San Francisco, CA); Adam Binsz (San Francisco, CA); Eugene Bistolas (Mountain View, CA); George Kedenburg (San Francisco, CA); Jenna Arnost (Kamas, UT)
Assignee: Hewlett-Packard Development Company, L.P.
G06F3/011G06F3/167G06V10/82G06V20/50G06V20/68G09G3/001G09G2354/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,411,542
App. No.
18/607,296
Granted
Sep 9, 2025
Kind
B2
Abstract

In an example method, a wearable multimedia device is worn by a user. Further, the device receives one or more communications during a first period of time; receives information regarding one or more events during the first period of time; receives a first spoken command from the user during a second period of time, where the second period of item is subsequent to the first period of time, and where the first spoken command includes a request to summarize the one or more communications and the one or more events; generates, using one or more of machine learning models, a summary of the one or more communications and the one or more events; and presents at least a portion of the summary to the user.

Claims (117)

1. A method comprising:

accessing, by a wearable multimedia device worn by a user, image data regarding an environment of the user, wherein the image data is generated using one or more image sensors of the wearable multimedia device;

determining, by the wearable multimedia device based on the image data, an item of interest in the environment of the user;

receiving, by the wearable multimedia device, a first user input from the user, wherein the first user input comprises a first command with respect to the item of interest, wherein the first user input is received using one or more microphones of the wearable multimedia device, and wherein the first command comprises a request for feedback regarding the item of interest;

performing, by the wearable multimedia device using one or more of machine learning models, the first command with respect to the item of interest, wherein performing the first command comprises generating first output data using the one or more machine learning models, and wherein generating the first output data comprises retrieving and aggregating product review information regarding the item of interest;

receiving, by the wearable multimedia device, a second user input from the user, wherein the second user input comprises a second command with respect to the item of interest, wherein the second user input is received using the one or more microphones of the wearable multimedia device, and wherein the second command comprises a request for information regarding the item of interest;

performing, by the wearable multimedia device using the one or more of machine learning models, the second command with respect to the item of interest, wherein performing the second command comprises generating second output data using the one or more machine learning models, and wherein the second output data comprises at least one of:

an identity of the item of interest, or

at least one of a make or a model of the item of interest; and

presenting, by the wearable multimedia device, at least a portion of the first output data and at least a portion of the second output data to the user.

2. A method comprising:

accessing, by a wearable multimedia device worn by a user, image data regarding an environment of the user, wherein the image data is generated using one or more image sensors of the wearable multimedia device;

determining, by the wearable multimedia device based on the image data, an item of interest in the environment of the user;

receiving, by the wearable multimedia device, a first user input from the user, wherein the first user input comprises a first command with respect to the item of interest, wherein the first user input is received using one or more microphones of the wearable multimedia device, and wherein the first command comprises a request for feedback regarding the item of interest;

performing, by the wearable multimedia device using one or more of machine learning models, the first command with respect to the item of interest, wherein performing the first command comprises generating first output data using the one or more machine learning models, and wherein generating the first output data comprises retrieving and aggregating product review information regarding the item of interest;

receiving, by the wearable multimedia device, a second user input from the user, wherein the second user input comprises a second command with respect to the item of interest, wherein the second user input is received using the one or more microphones of the wearable multimedia device, wherein the item of interest comprises an article of clothing, and wherein the second command comprises a request for information regarding the article of clothing;

performing, by the wearable multimedia device using the one or more of machine learning models, the second command with respect to the item of interest, wherein performing the second command comprises generating second output data using the one or more machine learning models, and wherein the second output data comprises at least one of:

at least one of a size or a fit of the article of clothing,

at least one of a color of a style of the article of clothing, or

an indication of a least one additional article of clothing to complement the article of clothing; and

presenting, by the wearable multimedia device, at least a portion of the first output data to the user.

3. A method comprising:

accessing, by a wearable multimedia device worn by a user, image data regarding an environment of the user, wherein the image data is generated using one or more image sensors of the wearable multimedia device;

determining, by the wearable multimedia device based on the image data, an item of interest in the environment of the user;

receiving, by the wearable multimedia device, a first user input from the user, wherein the first user input comprises a first command with respect to the item of interest, wherein the first user input is received using one or more microphones of the wearable multimedia device, and wherein the first command comprises a request for feedback regarding the item of interest;

performing, by the wearable multimedia device using one or more of machine learning models, the first command with respect to the item of interest, wherein performing the first command comprises generating first output data using the one or more machine learning models, and wherein generating the first output data comprises retrieving and aggregating product review information regarding the item of interest;

receiving, by the wearable multimedia device, a second user input from the user, wherein the second user input comprises a second command with respect to the item of interest, wherein the second user input is received using the one or more microphones of the wearable multimedia device, wherein the item of interest comprises a food item, and wherein the second command comprises a request for information regarding the food item;

performing, by the wearable multimedia device using the one or more of machine learning models, the second command with respect to the item of interest, wherein performing the second command comprises generating second output data using the one or more machine learning models, and wherein the second output data comprises at least one of:

a nutritional value of the food item,

one or more ingredients of the food item,

one or more allergens of the food item,

one or more serving sizes of the food item,

at least one of a place of origin or a place of production of the food item,

a recipe having the food item as an ingredient, or

an indication of a least one additional food item to complement the food item; and

presenting, by the wearable multimedia device, at least a portion of the first output data to the user.

4. The method of claim 3 , further comprising:

determining, based on the image data, that the user consumed the food item, and

storing, in a database, an indication that the user consumed the food item and the nutritional value of the food item, wherein the database represents a plurality of food items consumed by the user over a period of time.

5. The method of claim 4 , further comprising:

determining, based on the image data, a portion size of the food item, and

determining the nutritional value of the food item based on the portion size.

6. A method comprising:

accessing, by a wearable multimedia device worn by a user, image data regarding an environment of the user, wherein the image data is generated using one or more image sensors of the wearable multimedia device;

determining, by the wearable multimedia device based on the image data, an item of interest in the environment of the user;

receiving, by the wearable multimedia device, a first user input from the user, wherein the first user input comprises a first command with respect to the item of interest, wherein the first user input is received using one or more microphones of the wearable multimedia device, and wherein the first command comprises a request for feedback regarding the item of interest;

performing, by the wearable multimedia device using one or more of machine learning models, the first command with respect to the item of interest, wherein performing the first command comprises generating first output data using the one or more machine learning models, and wherein generating the first output data comprises retrieving and aggregating product review information regarding the item of interest;

receiving, by the wearable multimedia device, a second user input from the user, wherein the second user input comprises a second command with respect to the item of interest, wherein the second user input is received using the one or more microphones of the wearable multimedia device, and wherein the second command comprises a request for information regarding a purchase of the item of interest;

performing, by the wearable multimedia device using the one or more of machine learning models, the second command with respect to the item of interest, wherein performing the second command comprises generating second output data using the one or more machine learning models, and wherein the second output data comprises an indication of one or more retailers offering the item of interest for purchase; and

presenting, by the wearable multimedia device, at least a portion of the first output data to the user.

7. The method of claim 6 , wherein the second output data further comprises, for each of the one or more retailers, at least one of:

a distance of the retailer from the user,

a purchase price associated with the item of interest at that retailer,

an availability of the item of interest at the retailer, or

a shipping item associated with the item of interest by the retailer.

8. The method of claim 6 , further comprising:

receiving a third user input comprising a third command with respect to the item of interest, wherein the third user input comprises an indication to purchase the item of interest, and

initiating a purchase of the item of interest.

9. The method of claim 8 , wherein initiating the purchase of the item of interest comprises:

determining a payment method associated with the user,

selecting a retailer from among the one or more retailers, and

initiating the purchase of the item at the selected retailer using the payment method.

10. The method of claim 1 , wherein the one or more machine learning models comprise one or more computerized neural networks.

11. The method of claim 1 , wherein the image sensors comprise at least one of:

one or more cameras, or

one or more depth sensors.

12. The method of claim 1 , wherein presenting at least the portion of the first output data to the user comprises:

projecting, using a laser projector of the wearable multimedia device, a user interface on a surface of the user, wherein the user interface comprises at least the portion of the first output data.

13. The method of claim 12 , wherein the surface is a surface of a hand of a user.

14. The method of claim 12 , wherein the surface is a surface of a palm of a user.

15. The method of claim 1 , wherein presenting at least the portion of the first output data to the user comprises:

projecting, using a laser projector of the wearable multimedia device, at least a portion of the first output data on a surface of the item of interest.

16. The method of claim 1 , wherein presenting at least the portion of the first output data to the user comprises:

projecting, using a laser projector of the wearable multimedia device, at least a portion of the first output data on a surface in a proximity of the item of interest.

17. The method of claim 1 , wherein presenting at least the portion of the first output data to the user comprises:

generating, using one or more audio speakers of the wearable multimedia device, audio output comprising at least the portion of the first output data.

18. The method of claim 1 , further comprising:

receiving a third user input from the user, wherein the third user input comprises a third command with respect to the environment, wherein the third user input is received using the one or more microphones of the wearable multimedia device;

performing using the one or more of machine learning models, the third command with respect to the environment, wherein performing the third command comprises generating third output data using the one or more machine learning models; and

presenting at least a portion of the third output data to the user.

19. The method of claim 18 , wherein the third command comprises a request for a description of the environment, and

wherein the third output data comprises the description of the environment.

20. The method of claim 18 , wherein the third command comprises a request for contents of a sign in the environment, and

wherein the third output data comprises the contents of the sign.

21. The method of claim 18 , wherein the third command comprises a request for information regarding a business in the environment, and

wherein the third output data comprises the information regarding the business.

22. The method of claim 21 , wherein the information regarding the business comprises operating hours of the business.

23. A wearable multimedia device, comprising:

one or more image sensors;

one or more microphones;

an output device;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

accessing image data regarding an environment of a user wearing the wearable multimedia device, wherein the image data is generated using the one or more image sensors;

determining, based on the image data, an item of interest in the environment of the user;

receiving a first user input from the user, wherein the first user input comprises a first command with respect to the item of interest, wherein the first user input is received using the one or more microphones, and wherein the first command comprises a request for feedback regarding the item of interest;

performing the first command with respect to the item of interest, wherein performing the first command comprises generating first output data, and wherein generating the first output data comprises retrieving and aggregating product review information regarding the item of interest;

receiving a second user input from the user, wherein the second user input comprises a second command with respect to the item of interest, wherein the second user input is received using the one or more microphones, and wherein the second command comprises a request for information regarding the item of interest;

performing, using the one or more of machine learning models, the second command with respect to the item of interest, wherein performing the second command comprises generating second output data using the one or more machine learning models, and wherein the second output data comprises at least one of:

an identity of the item of interest, or

at least one of a make or a model of the item of interest; and

presenting, using the output device, at least a portion of the first output data to the user.

24. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

accessing, by a wearable multimedia device worn by a user, image data regarding an environment of the user, wherein the image data is generated using one or more image sensors of the wearable multimedia device;

determining, by the wearable multimedia device based on the image data, an item of interest in the environment of the user;

receiving, by the wearable multimedia device, a first user input from the user, wherein the first user input comprises a first command with respect to the item of interest, wherein the first user input is received using one or more microphones of the wearable multimedia device, and wherein the first command comprises a request for feedback regarding the item of interest;

performing, by the wearable multimedia device, the first command with respect to the item of interest, wherein performing the first command comprises generating first output data, and wherein generating the first output data comprises retrieving and aggregating product review information regarding the item of interest;

receiving a second user input from the user, wherein the second user input comprises a second command with respect to the item of interest, wherein the second user input is received using the one or more microphones of the wearable multimedia device, and wherein the second command comprises a request for information regarding the item of interest;

performing, using the one or more of machine learning models, the second command with respect to the item of interest, wherein performing the second command comprises generating second output data using the one or more machine learning models, and wherein the second output data comprises at least one of:

an identity of the item of interest, or

at least one of a make or a model of the item of interest; and

presenting, by the wearable multimedia device, at least a portion of the first output data to the user.

25. The method of claim 6 , the second output data further comprising:

for each of the one or more retailers, at least one of:

a distance of the retailer from the user, or

a shipping time associated with the item of interest by the retailer.

26. The method of claim 2 , wherein the article of clothing comprises a first type of clothing, and wherein the at least one additional article of clothing comprises a second type of clothing different from the first type of clothing.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 7, 2025
From: KOCIENDA, KENNETH LUKE; NULMAN, YANIR WELCZER; HASHEMI, LILYNAZ ARABI; CHAUDHRI, IMRAN A.; KOO, JANE EUNJUNG; BINSZ, ADAM; BISTOLAS, EUGENE; KEDENBURG, GEORGE LEWIS; ARNOST, JENNA VAIL
To: HUMANE, INC.
Reel/Frame 071960/0056 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 8, 2025
From: HUMANE, INC.
To: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.
Reel/Frame 071844/0747 →
Continuity (5)
Provisional Application 63496677 · Apr 17, 2023
Provisional Application 63454937 · Mar 27, 2023
Provisional Application 63453333 · Mar 20, 2023
Provisional Application 63453045 · Mar 17, 2023
Related Publication 20240310900A1 · Sep 19, 2024
References Cited (20)
US 11128636B1 · Jorasch et al. · 2021 [cited by applicant]
US 20130016070A1 · Starner · 2013 [cited by examiner]
US 20130336519A1 · Connor · 2013 [cited by applicant]
US 20160027329A1 · Jerauld · 2016 [cited by examiner]
US 20160360970A1 · Tzvieli et al. · 2016 [cited by applicant]
US 20180136465A1 · Chi · 2018 [cited by examiner]
US 20190037173A1 · Lee et al. · 2019 [cited by applicant]
US 20190179487A1 · Kong et al. · 2019 [cited by applicant]
US 20210373676A1 · Jorasch et al. · 2021 [cited by applicant]
US 20220415476A1 · Connor · 2022 [cited by examiner]
US 20240070993A1 · Tiku · 2024 [cited by examiner]
Kanel, “Sixth Sense Technology,” Thesis for the Bachelor Degree of Engineering in Information and Technology, Centria University of Applied Sciences, May 2014, 46 pages. [cited by applicant]
Mann et al., “Telepointer: Hands-Free Completely Self Contained Wearable Visual Augmented Reality without Headwear and without any Infrastructural Reliance,” IEEE Fourth International Symposium on Wearable Computers, At… [cited by applicant]
Mann, “Wearable Computing: A First Step Toward Personal Imaging,” IEEE Computer, Feb. 1997, 30(2):25-32. [cited by applicant]
Mann, “Wearable, tetherless computer-mediated reality,” American Association of Artificial Intelligence Technical Report, Feb. 1996, 62-69, 8 pages. [cited by applicant]
Metavision.com [online], “Sensularity with a Sixth Sense,” available on or before Apr. 7, 2015, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20170901072037/https://blog.metavision.com/professor-… [cited by applicant]
Mistry et al., “WUW—wear Ur world: a wearable gestural interface”, Proceedings of the 27th international conference Extended Abstracts on Human Factors in Computing Systems—CHI EA '09, Boston, MA, USA, Apr. 4-9, 2009, 6… [cited by applicant]
Shetty et al., “Sixth Sense Technology,” International Journal of Science and Research, Dec. 2014, 3(12):1068-1073. [cited by applicant]
Vaswani et al., “Attention is All You Need,” CoRR, submitted on Jun. 2017, arXiv:1706.03762, 15 pages. [cited by applicant]
International Search Report and Written Opinion in International Appln. No. PCT/US24/20280, mailed on Aug. 7, 2024, 14 pages. [cited by applicant]