IP Library › Granted Patent US 12,277,586
Granted Patent B2
US 12,277,586 · App. 17/481,218 · Granted Apr 15, 2025

Augmented reality systems and methods for purchasing

Inventors: Adrian Kaehler (Los Gatos, CA); Gary Bradski (Plantation, FL); Prasanna Krishnasamy (Plantation, FL); Doug Lee (Plantation, FL)
Assignee: Magic Leap, Inc.
G06Q30/0611G02B27/0172G06Q30/0283G06Q30/0629G06Q30/0643G06T19/006G02B2027/014
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,277,586
App. No.
17/481,218
Filed
Sep 21, 2021
Granted
Apr 15, 2025
Kind
B2
Art Unit
3688
USPC
705/26.4
Abstract

Disclosed herein is an augmented reality (AR) system that provides information about purchasing alternatives to a user who is about to purchase an item or product (e.g., a target product) in a physical retail location. In some variations, offers to purchase the product and/or an alternative product are provided by the merchant and/or competitors via the AR system. An offer negotiation server (ONS) aggregates offer data provided various external parties (EPs) and displays these offers to the user as the user is considering the purchase of a target product. In some variations, an AR system may be configured to facilitate the process of purchasing items at a retail location.

Claims (61)

1. A method comprising:

identifying, using an augmented reality device, a target product that is being considered for purchase by a user of the augmented reality device, wherein the identifying the target product using the augmented reality device comprises detecting that the user is viewing the target product at a retail location, wherein the detecting comprises:

acquiring, using an image sensor of the augmented reality device, an image of the user, and

executing instructions stored on a computer-readable medium to recognize a visual feature based on the image, the visual feature indicating a proximity between the target product and the user;

determining a target product price corresponding to the target product;

in accordance with the augmented reality device identifying the target product being considered for purchase, beginning a negotiation interval;

transmitting target product data from the augmented reality device to a remote server, wherein the target product data comprises target product identification data corresponding to the target product and further comprises the target product price;

receiving, by the augmented reality device, a purchase offer generated on the remote server, wherein the purchase offer comprises a message, wherein the generating the purchase offer on the remote server comprises identifying, using the product identification data, an alternate product and an alternate product offer price, and wherein the purchase offer is transmitted to the augmented reality device by the remote server in accordance with a determination, by the remote server, that the alternate product offer price is less than the target product price;

in response to receiving the purchase offer, displaying the purchase offer as augmented reality content on a display of the augmented reality device to the user;

tracking, by the image sensor of the augmented reality device, gestures of the user and executing instructions stored on a computer-readable medium to determine whether any of the tracked gestures corresponds to a command to accept the purchase offer;

determining an acceptance of the purchase offer in accordance with a recognition that at least one of the tracked gestures corresponds to the command to accept the purchase offer, and further in accordance with a location of the user determined via a location sensor of the augmented reality device;

in response to determining the acceptance of the purchase offer, terminating the negotiation interval;

detecting, by one or more sensors of the augmented reality device, that the target product has been picked up and is carried by the user;

generating, via the augmented reality device, a first alert upon detecting that the alternate product has been purchased during the negotiation interval and the target product is carried by the user, wherein:

a scanner at the retail location is configured to:

detect that the target product carried by the user is near an exit, and

transmit identification data of the target product to the remote server to determine whether the target item has been purchased by the user, and

the remote server is further configured to:

determine that the target product has not been purchased by the user, and

automatically charge a payment account of the user for the target product using pre-stored payment information; and

based on the automatically charging the payment account for the target product, receiving, via the augmented reality device, a second alert generated by the remote server, wherein the second alert indicates occurrence of a purchase transaction in accordance with the detection of the target product carried by the user near the exit and further in accordance with the determination that the target product had not been purchased by the user.

2. The method of claim 1 , wherein the displaying the purchase offer to the user comprises displaying the alternate product offer price to the user.

3. The method of claim 1 , wherein the target product data further comprises a geographic location of the target product.

4. The method of claim 3 , wherein the geographic location is represented by GPS coordinates.

5. The method of claim 1 , wherein the target product data comprises product category data.

6. The method of claim 1 , wherein the alternate product is the same as the target product.

7. The method of claim 1 , wherein the alternate product is in a same product category as the target product.

8. The method of claim 1 , wherein the displaying the purchase offer comprises displaying a graphic representing the alternate product and the alternate product offer price.

9. The method of claim 1 , further comprising transmitting a signal from the augmented reality device to the remote server indicating whether the user has accepted the purchase offer.

10. The method of claim 1 , wherein the remote server comprises a database of a plurality of computer-executable instruction sets, wherein each computer-executable instruction set is specific to a different product, wherein the generating the purchase offer comprises executing a computer-executable instruction set, of the plurality of computer-executable instruction sets, corresponding to the target product, and wherein the database is stored in non-volatile computer-readable medium.

11. The method of claim 10 , wherein one or more of the plurality of the product-specific computer-executable instruction sets have been transmitted to the remote server from a second remote server.

12. The method of claim 11 , wherein the second remote server is controlled by a merchant.

13. The method of claim 12 , further comprising transmitting offer decision data from the augmented reality device to the remote server indicating whether the user has accepted the purchase offer.

14. The method of claim 13 , further comprising generating a notification data structure from the remote server to the second remote server, wherein the notification data structure comprises offer decision data.

15. The method of claim 14 , wherein the notification data structure further comprises user-identification data.

16. The method of claim 15 , wherein the notification data structure further comprises user payment data.

17. The method of claim 1 , wherein the identifying the target product that is being considered for purchase by the user comprises detecting, via the augmented reality device, a distance between the user and the target product.

18. The method of claim 1 , wherein the identifying the target product that is being considered for purchase by the user comprises detecting, via the augmented reality device, a location of the target product relative to the retail location.

19. A system comprising:

a display; and

an augmented reality device having one or more processors configured to execute a method comprising:

identifying, using the augmented reality device, a target product that is being considered for purchase by a user of the augmented reality device, wherein the identifying the target product using the augmented reality device comprises detecting that the user is viewing the target product at a retail location, wherein the detecting comprises:

acquiring, using an image sensor of the augmented reality device, an image of the user, and

executing instructions stored on a computer-readable medium to recognize a visual feature based on the image, the visual feature indicating a proximity between the target product and the user;

determining a target product price corresponding to the target product;

in accordance with the augmented reality device identifying the target product being considered for purchase, beginning a negotiation interval;

transmitting target product data from the augmented reality device to a remote server, wherein the target product data comprises target product identification data corresponding to the target product and further comprises the target product price;

receiving, by the augmented reality device, a purchase offer generated on the remote server, wherein the purchase offer comprises a message, wherein the generating the purchase offer on the remote server comprises identifying, using the product identification data, an alternate product and an alternate product offer price, and wherein the purchase offer is transmitted to the augmented reality device by the remote server in accordance with a determination, by the remote server, that the alternate product offer price is less than the target product price;

in response to receiving the purchase offer, displaying the purchase offer as augmented reality content on a display of via the augmented reality device to the user;

tracking, by the image sensor of the augmented reality device, gestures of the user and executing instructions stored on a computer-readable medium to determine whether any of the tracked gestures corresponds to a command to accept the purchase offer;

determining an acceptance of the purchase offer in accordance with a recognition that at least one of the tracked gestures corresponds to the command to accept the purchase offer, and further in accordance with a location of the user determined via a location sensor of the augmented reality device;

in response to determining the acceptance of the purchase offer, terminating the negotiation interval;

detecting, by one or more sensors of the augmented reality device, that the target product has been picked up and is carried by the user;

generating, via the augmented reality device, a first alert upon detecting that the alternate product has been purchased during the negotiation interval and the target product is carried by the user, wherein:

a scanner at the retail location is configured to:

detect that the target product carried by the user is near an exit, and

transmit identification data of the target product to the remote server to determine whether the target item has been purchased by the user, and

the remote server is further configured to:

determine that the target product has not been purchased by the user, and

automatically charge a payment account of the user for the target product using pre-stored payment information; and

based on the automatically charging the payment account for the target product, receiving, via the augmented reality device, a second alert generated by the remote server, wherein the second alert indicates occurrence of a purchase transaction in accordance with the detection of the target product carried by the user near the exit and further in accordance with the determination that the target product had not been purchased by the user.

Assignments (4)
SECURITY INTEREST Recorded Oct 31, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073422/0549 →
SECURITY INTEREST Recorded Oct 28, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073388/0027 →
SECURITY INTEREST Recorded Oct 24, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073255/0581 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2023
From: KAEHLER, ADRIAN; BRADSKI, GARY; KRISHNASAMY, PRASANNA; LEE, DOUG
To: MAGIC LEAP, INC.
Reel/Frame 064570/0014 →
Continuity (4)
Continuation 15192724 · Jun 24, 2016
Provisional Application 62195184 · Jul 21, 2015
Provisional Application 62184150 · Jun 24, 2015
Related Publication 20220005095A1 · Jan 6, 2022
References Cited (92)
US 4852988A · Velez · 1989 [cited by applicant]
US 6433760B1 · Vaissie · 2002 [cited by applicant]
US 6491391B1 · Blum et al. · 2002 [cited by applicant]
US 6847336B1 · Lemelson · 2005 [cited by applicant]
US 6943754B2 · Aughey · 2005 [cited by applicant]
US 6977776B2 · Volkenandt et al. · 2005 [cited by applicant]
US 7347551B2 · Fergason et al. · 2008 [cited by applicant]
US 7488294B2 · Torch · 2009 [cited by applicant]
US 8235529B1 · Raffle · 2012 [cited by applicant]
US 8595073B2 · Sprigg et al. · 2013 [cited by applicant]
US 8611015B2 · Wheeler · 2013 [cited by applicant]
US 8638498B2 · Bohn et al. · 2014 [cited by applicant]
US 8696113B2 · Lewis · 2014 [cited by applicant]
US 8929589B2 · Publicover et al. · 2015 [cited by applicant]
US 9010929B2 · Lewis · 2015 [cited by applicant]
US 9274338B2 · Robbins et al. · 2016 [cited by applicant]
US 9292973B2 · Bar-zeev et al. · 2016 [cited by applicant]
US 9323325B2 · Perez et al. · 2016 [cited by applicant]
US 9665881B1 · Ward · 2017 [cited by applicant]
US 9720505B2 · Gribetz et al. · 2017 [cited by applicant]
US 10013053B2 · Cederlund et al. · 2018 [cited by applicant]
US 10025379B2 · Drake et al. · 2018 [cited by applicant]
US 10664795B1 · Worley · 2020 [cited by examiner]
US 11164227B2 · Kaehler et al. · 2021 [cited by applicant]
US 20030030597A1 · Geist · 2003 [cited by applicant]
US 20040267611A1 · Hoerenz · 2004 [cited by applicant]
US 20060023158A1 · Howell et al. · 2006 [cited by applicant]
US 20070210155A1 · Swartz et al. · 2007 [cited by applicant]
US 20080103939A1 · Gibb · 2008 [cited by examiner]
US 20110211056A1 · Publicover et al. · 2011 [cited by applicant]
US 20110213664A1 · Osterhout · 2011 [cited by applicant]
US 20120021806A1 · Maltz · 2012 [cited by applicant]
US 20120154557A1 · Perez et al. · 2012 [cited by applicant]
US 20120233003A1 · Calman · 2012 [cited by examiner]
US 20120280040A1 · Carney et al. · 2012 [cited by applicant]
US 20130077147A1 · Efimov · 2013 [cited by applicant]
US 20140172570A1 · y Arcas · 2014 [cited by examiner]
US 20140195918A1 · Friedlander · 2014 [cited by applicant]
US 20140279422A1 · Holman et al. · 2014 [cited by applicant]
US 20150168731A1 · Robbins · 2015 [cited by applicant]
US 20150242943A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150324881A1 · Ouimet · 2015 [cited by examiner]
US 20150348101A1 · Junger · 2015 [cited by applicant]
US 20150363816A1 · Poglitsch · 2015 [cited by examiner]
CA 2316473A1 · 2001 [cited by applicant]
CA 2362895A1 · 2002 [cited by applicant]
CA 2388766A1 · 2003 [cited by applicant]
EP 2544138A1 · 2013 [cited by applicant]
JP 2013077328A · 2013 [cited by applicant]
JP 2013521576A1 · 2013 [cited by applicant]
JP 2014520341A · 2014 [cited by applicant]
JP 2015111358A · 2015 [cited by applicant]
KR 20110128308A · 2011 [cited by applicant]
WO 2011122654A1 · 2011 [cited by applicant]
WO 2011106798A1 · 2011 [cited by applicant]
WO 2012177382A2 · 2012 [cited by applicant]
WO 2016210354A1 · 2016 [cited by applicant]
Borison, Rebecca, Valpak integrates augmented reality for easier coupon discovery, Oct. 18, 2013, retaildrive.com, accessed at [https://www.retaildive.com/ex/mobilecommercedaily/valpak-integrates-augmented-reality-in-ap… [cited by examiner]
Canadian Office Action dated Oct. 11, 2022, for CA Application No. 2,989,894, three pages. [cited by applicant]
Jacob, R. “Eye Tracking in Advanced Interface Design”, Virtual Environments and Advanced Interface Design, Oxford University Press, Inc. (Jun. 1995). [cited by applicant]
Japanese Notice of Allowance mailed Nov. 7, 2022, for JP Application No. 2021-005703, with English translation, 6 pages. [cited by applicant]
Rolland, J. et al., “High-resolution inset head-mounted display”, Optical Society of America, vol. 37, No. 19, Applied Optics, (Jul. 1, 1998). [cited by applicant]
Tanriverdi, V. et al. (Apr. 2000), “Interacting With Eye Movements In Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA 02155, USA, Proceedings of the SIGCHI… [cited by applicant]
Yoshida, A. et al., “Design and Applications of a High Resolution Insert Head Mounted Display”, (Jun. 1994). [cited by applicant]
Azuma, Ronald T. (Aug. 1997). “A Survey of Augmented Reality,” In Presence: Teleoperators and Virtual Environments 6, 4, Hughes Research Laboratories, Malibu, CA, located at: https://web.archive.org/web/20010604100006/h… [cited by applicant]
Bimber, Oliver et al. (2005). “Spatial Augmented Reality: Merging Real and Virtual Worlds,” A. K. Peters, Ltd., Wellesley, MA. [cited by applicant]
Chinese Office Action dated Apr. 22, 2021, for CN Application No. 201680047017.X, with English translation, 23 pages. [cited by applicant]
Chinese Office Action dated Oct. 27, 2021, for CN Application No. 201680047017.X, with English translation, 7 pages. [cited by applicant]
Examination Report dated Apr. 6, 2021, for AU Application No. 2016284700, five pages. [cited by applicant]
Extended European Search Report dated Oct. 25, 2021.00, for EP Application No. 21188663.5, thirteen pages. [cited by applicant]
Final Office Action mailed Aug. 20, 2020, for U.S. Appl. No. 15/192,724, filed Jun. 24, 2016, 24 pages. [cited by applicant]
Indian Office Action dated May 5, 2021, for IN Application No. 201847002916, with English translation, six pages. [cited by applicant]
Israeli Office Action dated Sep. 23, 2020, for IL Patent Application No. 256344, with English translation, seven pages. [cited by applicant]
Non-Final Office Action mailed Apr. 8, 2019, for U.S. Appl. No. 15/192,724, filed Jun. 24, 2016, 20 pages. [cited by applicant]
Notice of Allowance mailed Jun. 23, 2021, for U.S. Appl. No. 15/192,724, filed Jun. 24, 2016, 17 pages. [cited by applicant]
Notice of Allowance, mailed May 20, 2021, for JP Application No. 2017-566275, with English translation, six pages. [cited by applicant]
Notice of Final Rejection, dated Sep. 16, 2020, for JP Application No. 2017-566275, with English translation, 6 pages. [cited by applicant]
Notice of Reasons for Rejection, dated Jan. 15, 2020, for JP Application No. 2017-566275, with English translation, 18 pages. [cited by applicant]
International Search Report for PCT Application No. PCT/US2016/039401, mailed Nov. 3, 2016, filed on Jun. 24, 2016, five pages. [cited by applicant]
Korean Office Action dated Jul. 24, 2023, for KR Application No. 10-2018-7002200, with English translation, 16 pages. [cited by applicant]
Indian Office Action dated Sep. 15, 2022, for IN Application No. 202148050391, with English translation, seven pages. [cited by applicant]
Examination Report dated Apr. 5, 2022, for Australian Application No. 2016284700, nine pages. [cited by applicant]
Japanese office action mailed Mar. 31, 2022, for JP Application No. 2021005703, with English translation, 18 pages. [cited by applicant]
Examination Report dated Jul. 14, 2022, for New Zealand Application No. 777443, three pages. [cited by applicant]
Examination Report dated Jul. 14, 2022, for New Zealand Application No. 777449, three pages. [cited by applicant]
International Preliminary Report on Patentability mailed Jan. 4, 2018, for PCT Application No. PCT/US2016/039401, filed Jun. 24, 2016, eight pages. [cited by applicant]
Examination Report dated Jul. 14, 2022, for New Zealand Application No. 738603, three pages. [cited by applicant]
Chinese Notice of Allowance dated Mar. 2, 2022 for CN Application No. 201680047017.X, with English translation, 4 pages. [cited by applicant]
Israeli Notice of Allowance dated Feb. 24, 2022, for IL Patent Application No. 256344, with English translation, six pages. [cited by applicant]
Examination Report dated Dec. 6, 2021, for Australian Application No. 2016284700, five pages. [cited by applicant]
Korean Notice of Allowance dated May 7, 2024, for KR Application No. 10-2024-7012278, with English translation, four pages. [cited by applicant]
Korean Notice of Allowance dated Feb. 8, 2024, for KR Application No. 10-2018-7002200, with English translation, four pages. [cited by applicant]
Cited By (1)
US 12,591,923