IP Library Granted Patent US 11,656,336
Granted Patent B2
US 11,656,336 · App. 17/119,312 · Granted May 23, 2023

Advanced gaming and virtual reality control using radar

Inventors: Patrick M. Amihood (Palo Alto, CA); Ivan Poupyrev (Sunnyvale, CA)
Assignee: Google LLC
G06F3/017A63F13/21A63F13/24G01S7/4004G01S7/41G01S7/412G01S7/415G01S13/56G01S13/66G01S13/86G01S13/867G01S13/88G01S13/888G01S13/90G01S13/904G06F3/011G06F3/04815G06F16/245G06F21/32G06F21/6245G06K9/629G06K9/6254G06K9/6255G06K9/6262G06K9/6288G06N20/00G06V20/64G06V40/28H04Q9/00H04W4/80H04W16/28A63F2300/8082G01S13/865G01S13/931G01S19/42G01S2013/9322G06F1/163G06F3/0346G06F3/0484G06F3/165G06F2203/0384G06F2221/2105G06T7/75G08C17/02G08C2201/93H04Q2209/883
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,656,336
App. No.
17/119,312
Granted
May 23, 2023
Kind
B2
Abstract

Techniques are described herein that enable advanced gaming and virtual reality control using radar. These techniques enable small motions and displacements to be tracked, even in the millimeter or submillimeter scale, for user control actions even when those actions are optically occluded or obscured.

Claims (72)

1. A computer-implemented method for controlling a gaming or virtual-reality application, the computer-implemented method comprising:

receiving a radar signal representing reflections of a radar field off a first point and a second point over time, at least one of the first point or the second point being on a portion of a user;

determining, based on the radar signal, relative Doppler-frequencies between the first point and the second point over time;

determining, based on the relative Doppler-frequencies, a user-control action for the gaming or virtual-reality application;

directing, based on the user-control action, the gaming or virtual-reality application to perform an action;

determining, based on the radar signal, locations of at least one of the first point or the second point over time; and

directing, based on the determined locations, the gaming or virtual-reality application to alter a display associated with the gaming or virtual-reality application.

2. The computer-implemented method of claim 1 :

wherein the radar signal further represents reflections of the radar field off one or more other points over time; and

further comprising:

determining, based on the radar signal, micro-Doppler centroids over time; and

distinguishing, based on the micro-Doppler centroids, the first point and the second point from the other points over time.

3. The computer-implemented method of claim 1 , further comprising spatially resolving the first point and the second point over time by determining:

radial distances for the first point and the second point based on respective time delays determined from the radar signal;

radial velocities for the first point and the second point based on respective Doppler frequencies determined from the radar signal; and

reflected energies for the first point and the second point determined from the radar signal.

4. The computer-implemented method of claim 1 , wherein the portion of the user is a hand of the user.

5. The computer-implemented method of claim 4 , wherein the first point and the second point are on the hand of the user.

6. The computer-implemented method of claim 4 , wherein the altering the display in the gaming or virtual-reality application comprises representing the hand of the user within the gaming or virtual-reality application.

7. The computer-implemented method of claim 4 , wherein the altering the display in the gaming or virtual-reality application comprises representing an object that is being held by the hand of the user within the gaming or virtual-reality application.

8. The computer-implemented method of claim 7 , wherein the object does not have control or communication capabilities.

9. The computer-implemented method of claim 8 , wherein the object is a mock game controller.

10. The computer-implemented method of claim 1 , further comprising determining, based on the relative Doppler frequencies, relative velocities between the first point and the second point over time, wherein the user-control action is determined based further on the relative velocities.

11. The computer-implemented method of claim 10 , further comprising integrating the relative velocities to determine relative displacements between the first point and the second point over time, wherein the user-control action is determined based further on the relative displacements.

12. An apparatus comprising:

at least one processor;

a radar system configured to:

provide a radar field over time; and

receive a radar signal representing reflections of the radar field off points over time; and

at least one computer-readable storage medium having instructions stored thereon that, responsive to execution by the processor, cause the processor to

determine, based on the radar signal, relative Doppler frequencies between a first point and a second point of the points over time, at least one of the first point or the second point being on a portion of a user;

determine, based on the relative Doppler frequencies, a user-control action; and

direct, based on the user-control action, a game application or virtual-reality program to perform an action;

determine, based on the radar signal, locations of at least one of the first point or the second point relative to the apparatus over time; and

direct, based on the determined locations, the game application or the virtual-reality program to alter a display associated with the game or virtual-reality application.

13. The apparatus of claim 12 , wherein the apparatus is configured as smart glasses, a smart watch, a virtual or augmented-reality system, a gaming system, a mobile phone, a tablet, a laptop, or a desktop computer.

14. The apparatus of claim 12 , wherein at least one of the user-control action or the locations is passed to the game application or the virtual-reality program via at least one application programming interface (API).

15. The apparatus of claim 12 , wherein:

the locations correspond to a hand of the user; and

the passing the locations to the game application or the virtual-reality program is further effective to enable the game application or the virtual-reality program to display a representation of the hand of the user within the game application or the virtual-reality program.

16. The apparatus of claim 12 , wherein:

the locations correspond to an object being held by the user; and

the passing the locations to the game application or the virtual-reality program is further effective to enable the game application or the virtual-reality program to display a representation of the object within the game application or the virtual-reality program.

17. The apparatus of claim 16 , wherein the object does not have control or communication capabilities.

18. The apparatus of claim 17 , wherein the object is a mock game controller.

19. The apparatus of claim 12 , wherein the control recognition module is further configured to determine the user-control action based on relative velocities between the first point and the second point over time, the relative velocities based on the relative Doppler frequencies.

20. The apparatus of claim 19 , wherein the control recognition module is further configured to determine the user-control action based on relative displacements between the first point and the second point over time, the relative displacements based on an integral of the relative velocities over time.

21. The apparatus of claim 12 , further comprising the display, and wherein:

the computer-readable storage medium further includes instructions for the game application or the virtual-reality program; and

the instructions to direct the game application or the virtual-reality program to alter the display are transmitted within the apparatus between one or more of the at least one processor, the at least one computer-readable storage medium, or the display.

22. At least one computer-readable storage medium having instructions stored thereon that, responsive to execution by at least one processor, cause the processor to:

receive a radar signal representing reflections of the radar field off a first point and a second point over time, at least one of the first point or the second point being on a portion of a user;

determine, based on the radar signal, relative Doppler frequencies between the first point and the second point over time;

determine, based on the relative Doppler frequencies, a user-control action for a gaming or virtual-reality application;

direct, based on the user-control action, the gaming or virtual-reality application to perform an action;

determine locations of at least one of the first point or the second point over time; and

direct, based on the determined locations, the gaming or virtual-reality application to alter a display associated with the gaming or virtual-reality application.

23. The computer-readable storage medium of claim 22 , wherein:

the locations correspond to a hand of the user; and

the passing the locations to the game application or the virtual-reality program is effective to enable the game application or the virtual-reality program to display a representation of the hand of the user within the game application or the virtual-reality program.

24. The computer-readable storage medium of claim 22 , wherein:

the locations correspond to an object being held by the user; and

the passing the locations to the game application or the virtual-reality program is effective to enable the game application or the virtual-reality program to display a representation of the object within the game application or the virtual-reality program.

25. The computer-readable storage medium of claim 24 , wherein the object does not have control or communication capabilities.

26. The computer-readable storage medium of claim 25 , wherein the object is a mock game controller.

27. The computer-readable storage medium of claim 22 , wherein:

the instructions further cause the processor to determine, based on the relative Doppler frequencies, relative velocities between the first point and the second point over time; and

the user-control action is determined based further on the relative velocities.

28. The computer-readable storage medium of claim 27 , wherein:

the instructions further cause the processor to integrate the relative velocities to determine relative displacements between the first point and the second point over time; and

wherein the user-control action is determined based further on the relative displacements.

29. The computer-readable storage medium of claim 22 , wherein the computer-readable storage medium further includes instructions for the game application or the virtual-reality program.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 11, 2020
From: AMIHOOD, PATRICK M.; POUPYREV, IVAN
To: GOOGLE INC.
Reel/Frame 054618/0106 →
CHANGE OF NAME Recorded Dec 11, 2020
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 054706/0801 →
Continuity (4)
Continuation 16380245 · Apr 10, 2019
Continuation 15286495 · Oct 5, 2016
Provisional Application 62237975 · Oct 6, 2015
Related Publication 20210096653A1 · Apr 1, 2021
Cited By (1)
US 12,340,028