Fine-motion virtual-reality or augmented-reality control using radar
This document describes techniques for fine-motion virtual-reality or augmented-reality control using radar. These techniques enable small motions and displacements to be tracked, even in the millimeter or sub-millimeter scale, for user control actions even when those actions are small, fast, or obscured due to darkness or varying light. Further, these techniques enable fine resolution and real-time control, unlike conventional RF-tracking or optical-tracking techniques.
1. A system comprising:
at least one processor; and
computer-readable storage media comprising instructions that, when executed by the at least one processor, cause the system to:
cause a display to present a virtual-reality (VR) or augmented-reality (AR) object within a VR world or an AR viewport;
track a user interaction with the VR/AR object over time, the tracking comprising:
causing a radar system to transmit a radar field over time;
receiving, from the radar system, a radar signal representing a superposition of reflections of the radar field off two or more spatially separated points of a hand of a user that performs the user interaction over time;
determining, based on the received radar signal, relative velocities between the spatially separated points of the hand over time;
determining, based on the relative velocities, movement of the hand of the user over time; and
determining, based on the received radar signal, locations of the hand of the user over time,
the tracking based on the movement and the locations of the hand of the user over time; and
cause the display to change the presentation of the VR/AR object in real time according to the user interaction.
2. The system of claim 1 , further comprising the display.
3. The system of claim 1 , further comprising the radar system.
4. The system of claim 3 , wherein the radar system is configured to transmit the radar field and receive the radar signal through fabric.
5. The system of claim 4 , wherein the system is configured to be disposed in a pocket or bag of a user performing the user interaction.
6. The system of claim 3 , further comprising the display.
7. The system of claim 6 , wherein the system is comprised by smart glasses or VR goggles.
8. The system of claim 6 , wherein the system is comprised by a smartphone.
9. The system of claim 1 , wherein:
the VR/AR object comprises a number pad or virtual keyboard; and
the user interaction is a button press on the number pad or the virtual keyboard.
10. The system of claim 1 , wherein:
the VR/AR object is a button, knob, or slider; and
the user interaction is a press of the button, a turn of the knob, or a movement of the slider.
11. The system of claim 1 , wherein:
the VR/AR object is a steering wheel; and
the user interaction is a turn of the steering wheel.
12. The system of claim 1 , wherein the user interaction is a selection corresponding to the VR/AR object.
13. The system of claim 1 , wherein:
the causing of the display to present the VR/AR object comprises causing the display to present the VR/AR object superimposed on a real-world object;
the user interaction corresponds to the real-world object; and
the causing of the display to change the presentation of the VR/AR object comprises causing the display to change the presentation of the VR/AR object according to the user interaction with the real-world object.
14. The system of claim 1 , wherein the causing of the display to change the presentation of the VR/AR object comprises causing the display to present a visual indication of the user interaction with the VR/AR object.
15. The system of claim 14 , wherein the causing of the display to present the visual indication comprises causing the display to alter an appearance of the VR/AR object.
16. The system of claim 15 , wherein the causing of the display to alter the appearance comprises causing the display to change a color of a portion of the VR/AR object.
17. The system of claim 1 , wherein:
the radar signal further represents reflections of the radar field off another spatially separated point over time; and
the tracking of the user interaction is further based on the reflections of the radar field off the other spatially separated point.
18. The system of claim 17 , wherein the other spatially separated point is not on the hand of the user.
19. The system of claim 1 , wherein the spatially separated points are on respective fingers of the hand of the user.
20. The system of claim 19 , wherein the movement comprises a first finger crossing a second finger.
21. The system of claim 1 , wherein at least one of the spatially separated points is visually obscured from the system.
22. The system of claim 21 , wherein at least one of the spatially separated points is visually obscured by another of the spatially separated points.
23. The system of claim 1 , wherein the tracking of the user interaction is on a millimeter or sub-millimeter scale.
24. The system of claim 1 , wherein the instructions further cause the system to pass a control input corresponding to the user interaction effective to control the VR world, the AR viewport, or an application or device external to the VR world or the AR viewport.
25. The system of claim 1 , wherein the VR/AR object comprises a virtual controller.
26. A system comprising:
at least one processor; and
computer-readable storage media comprising instructions that, when executed by the at least one processor, cause the system to:
cause a display to present a virtual-reality (VR) or augmented-reality (AR) object within a VR world or an AR viewport;
track a user interaction with the VR/AR object over time, the tracking comprising:
causing a radar system to transmit a radar field over time;
receiving, from the radar system, a radar signal representing a superposition of reflections of the radar field off two or more spatially separated points of a portion of a user that performs the user interaction over time;
distinguishing the reflections of the spatially separated points by determining respective Doppler centroids within the radar signal for the spatially separated points;
spatially resolving, based on the respective Doppler centroids, the spatially separated points over time; and
determining, based on the spatially resolving, locations and movement of the portion of the user over time, the tracking based on the determined locations and movement of the portion of the user over time; and
cause the display to change the presentation of the VR/AR object in real time according to the user interaction.
27. The system of claim 26 , further comprising the display.
28. The system of claim 26 , further comprising the radar system.
29. The system of claim 28 , wherein the radar system is configured to transmit the radar field and receive the radar signal through fabric.
30. The system of claim 29 , wherein the system is configured to be disposed in a pocket or bag of a user performing the user interaction.
31. The system of claim 28 , further comprising the display.
32. The system of claim 31 , wherein the system is comprised by smart glasses or VR goggles.
33. The system of claim 31 , wherein the system is comprised by a smartphone.
34. The system of claim 26 , wherein:
the VR/AR object comprises a number pad or virtual keyboard; and
the user interaction is a button press on the number pad or the virtual keyboard.
35. The system of claim 26 , wherein:
the VR/AR object is a button, knob, or slider; and
the user interaction is a press of the button, a turn of the knob, or a movement of the slider.
36. The system of claim 26 , wherein:
the VR/AR object is a steering wheel; and
the user interaction is a turn of the steering wheel.
37. The system of claim 26 , wherein the user interaction is a selection corresponding to the VR/AR object.
38. The system of claim 26 , wherein:
the causing of the display to present the VR/AR object comprises causing the display to present the VR/AR object superimposed on a real-world object;
the user interaction corresponds to the real-world object; and
the causing of the display to change the presentation of the VR/AR object comprises causing the display to change the presentation of the VR/AR object according to the user interaction with the real-world object.
39. The system of claim 26 , wherein the causing of the display to change the presentation of the VR/AR object comprises causing the display to present a visual indication of the user interaction with the VR/AR object.
40. The system of claim 39 , wherein the causing of the display to present the visual indication comprises causing the display to alter an appearance of the VR/AR object.
41. The system of claim 40 , wherein the causing of the display to alter the appearance comprises causing the display to change a color of a portion of the VR/AR object.
42. The system of claim 26 , wherein:
the radar signal further represents reflections of the radar field off another spatially separated point over time; and
the tracking of the user interaction is further based on the reflections of the radar field off the other spatially separated point.
43. The system of claim 42 , wherein the other spatially separated point is not on the portion of the user.
44. The system of claim 26 , wherein the spatially separated points are on respective fingers of a hand of the user.
45. The system of claim 44 , wherein the movement comprises a first finger crossing a second finger.
46. The system of claim 26 , wherein at least one of the spatially separated points is visually obscured from the system.
47. The system of claim 46 , wherein at least one of the spatially separated points is visually obscured by another of the spatially separated points.
48. The system of claim 26 , wherein the tracking of the user interaction is on a millimeter or sub-millimeter scale.
49. The system of claim 26 , wherein the instructions further cause the system to pass a control input corresponding to the user interaction effective to control the VR world, the AR viewport, or an application or device external to the VR world or the AR viewport.
50. The system of claim 26 , wherein the VR/AR object comprises a virtual controller.